Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017
Deterministic Optimization and Design
Class Notes for Civil Engineering 153
Department of Civil and Environmental Engineering University of California, Davis
Jay R. Lund
February 2016
© 2017 Jay R. Lund
"... engineers who can do only the best of what has already been done will be replaced by a computer system." Daniel C. Drucker Engineering Education, Vol. 81, No. 5, July/August 1991, p. 480.
"Good management is the art of making problems so interesting and their solutions so constructive that everyone wants to get to work and deal with them." Paul Hawken
Acknowledgments: These notes have improved steadily over the years from corrections and suggestions by numerous students, TAs, and substitute instructors. Particularly useful improvements have come from Ken Kirby, Orit Kalman, Min Yip, Stacy Tanaka, David Watkins, and Rui Hui, with many others contributing as well.
1 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Table of Contents Some Thoughts on Optimization ...... 3 Introduction/Overview ...... 5 Design of A Bridge Over A Gorge ...... 7 Systems Analysis ...... 10 Steps to Solving a Problem Using Optimization ...... 12 Unconstrained Optimization Using Calculus ...... 16 Analytical vs. Search methods ...... 21 Lagrange Multipliers ...... 24 Alternative Method for Inequality Constraints ...... 31 Mathematical Programming ...... 33 Karush-Kuhn-Tucker (KKT) Conditions for an Optimal Solution ...... 36 Linear Programming ...... 37 The Simplex Method: A Graphical Interpretation ...... 41 The Simplex Tableau ...... 42 The Simplex Method - An Algorithm ...... 44 Some Simplex Details ...... 48 Implementing the Big-M Method ...... 52 Sensitivity Analysis in Linear Programming ...... 54 Duality...... 58 Example optimization: California Hydropower ...... 61 Transportation Problems ...... 61 Scheduling Problems ...... 63 LP Solution of Non-Linear Objectives with Linear Constraints ...... 65 Integer Programming and Mixed Integer-Linear Programming ...... 67 Summary of the BIP Branch and Bound Algorithm ...... 69 More Non-Linear Objective Functions with Linear Constraints Using Integer-Linear Programming...... 72 Computational Aspects ...... 73 Dynamic Programming (DP) ...... 74 More DP Applications ...... 81 Non-Linear Programming ...... 84 Genetic Algorithms/Evolutionary Algorithms ...... 93 The Final Days ...... 100 Appendices ...... 101 PROBLEM FORMULATION: SOME GUIDELINES ...... 103 INTERPRETATION OF RESULTS: SOME GUIDELINES ...... 104 INTRODUCTION TO SOLVING LINEAR PROGRAMS USING LINDO ...... 105 THE SIMPLEX ALGORITHM ...... 106 Selected Bibliography ...... 107
2 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Some Thoughts on Optimization
"All models are wrong, but some are useful." - G.E.P. Box (1979), Robustness in Statistics.
"Operations research is the art of giving bad answers to problems to which otherwise worse answers are given." - T.L. Saaty (1959), Mathematical Methods of Operations Research, p. 3.
"What would life be without arithmetic, but a scene of horrors." - Revd. Sydney Smith, 1835
“The purpose of mathematical programming is insight, not numbers” – A.M. Geoffrion 1978
"Decision analysis separates a large-scale problem into its sub-parts, each of which is simpler to manipulate and diagnose. After the separate elements are carefully examined, the results are synthesized to give insights into the original problem." - Harvey Wagner (1975), Principles of Operations Research.
"The adoption of operations research calls for an act of faith in the potential benefits of a systematic approach to decision-making." - Harvey Wagner (1975), Principles of Operations Research.
"Even when quantitative analysis is of central importance for a managerial decision process, an operations- research-oriented system never supplies all the information required for action, no matter how sophisticated the system's design. Furthermore, a truly successful implementation of an operations research system must apply behavioral as well as mathematical science, because the resultant system must interact with human beings. And finally, the very process of constructing an operations research system involves the exercise of judgement in addition to the logical manipulation of symbols and data." - Harvey Wagner (1975), Principles of Operations Research.
"Le mieux est l'ennemi du bien." The best is the enemy of the good. - Voltaire
“A great craftsman does not put aside the plumb-line for the benefit of a clumsy carpenter.” Mencius (China, c. 300 B.C., Book VII, Part A, 41)
"Life must be lived forwards, but it can only be understood backwards" - Soren Kierkegaard
"A man's gotta know his limitations." - Clint Eastwood
"A problem well put is a problem half solved." - Anon.
“The bottom line for mathematicians is that the architecture has to be right. In all the mathematics that I did, the essential point was to find the right architecture. It's like building a bridge. Once the main lines of the structure are right, then the details miraculously fit. The problem is the overall design.” Freeman Dyson, "Freeman Dyson: Mathematician, Physicist, and Writer" interview with Donald J. Albers, The College Mathematics Journal, vol 25, no. 1, January 1994.
“An expert problem solver must be endowed with two incompatible qualities, a restless imagination and a patient pertinacity.” Howard W. Eves in Mathematical Circles, Boston: Prindle, Weber and Schmidt, 1969.
“The errors of definitions multiply themselves according as the reckoning proceeds; and lead men into absurdities, which at last they see but cannot avoid, without reckoning anew from the beginning.” Thomas Hobbes, in J. R. Newman (ed.) The World of Mathematics, New York: Simon and Schuster, 1956.
3 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017
“In order to translate a sentence from English into French two things are necessary. First, we must understand thoroughly the English sentence. Second, we must be familiar with the forms of expression peculiar to the French language. The situation is very similar when we attempt to express in mathematical symbols a condition proposed in words. First, we must understand thoroughly the condition. Second, we must be familiar with the forms of mathematical expression. George Polyá, How to Solve It. Princeton: Princeton University Press. 1945.
People are often unhappy with complex system designs, but other choices are often worse.
xkcd.com
Solving by enumeration "They always bury [treasure] under a haunted house, or on an island, or under a dead tree that's got one limb sticking out. Well, we've tried Jackson Island a little, and we can try it again some time; and there's lots of dead-limb trees - dead loads of em. Is it under all of them? How you talk! No! Then how you going to know which one to go for? Go for all of em. Why, Tom, it'll take all summer." - Mark Twain, The Adventures of Tom Sawyer, 1876.
4 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Introduction/Overview
What is "Deterministic Optimization"?
1) Optimization = A process of finding the "best" solution or design to a problem
2) Deterministic = Problems or systems that are well known; assume no probability or uncertainty
Optimization: - finding the "best" solution or design to a problem
1. What do we mean by the "best"? cost, performance, aesthetics, social or individual "well-being", etc.
The objective implied by "best" will vary with the problem and designer/client.
Example: The "best" use of Mono Lake for L.A. might be to let it dry up. The "best" use of Mono Lake for Davis is for recreation.
2. Why can't Mono Lake serve both intensive water supply and recreation? The use of Mono Lake is constrained by:
1) limited water availability 2) the ecosystem of Mono Lake
Can't make everybody perfectly happy, but have to do the "best" within constraints.
Optimization selects the "best" decision from a constrained situation.
Obviously, the "best" solution is driven by your objectives for solving a problem.
A practical insight from optimization, enumeration theory:
xkcd.com 5 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Course Mechanics
Everyone needs computer programming for this course. Use whichever you are most comfortable with. Get comfortable with one way to program, you'll be using it a lot!
-) Use of the computer for this class: after ~3rd week you will be using it a lot. -) Use the computer to help you solve problems for other classes. Especially problems with trial-and- error solutions. "Computers should work; people should think." -- But before this can happen you have to become familiar with getting computers to do your calculations. -) Use of a computer language is required for this course (FORTRAN, BASIC, PASCAL, C, Visual Basic, MATLAB, etc., whatever you are most comfortable with). This is unavoidable for Project No. 2. For those uncomfortable with programming, I recommend Visual Basic for Applications tutorials on the class web site, and elsewhere.
-) Weekly Homework or Projects (eight total)
Homework dwells more on techniques and methods. Projects make you think a problem through more and so have more individual answers, and generally require more work. Almost all assignments will require use of a computer.
-) One mid-term -) One final
The article passed out gives a useful philosophical perspective on this class, and how this class should fit into your larger education as an engineer.
Course Objectives: Classical Optimization Theory Formulates All Problems into the form:
Maximize or Minimize: an objective function
Subject To: a set of mathematical constraints
This becomes the standard form for all problem statements for this class.
The objectives of this class are:
a) to help you formulate engineering and other problems in this format, b) to give you some mathematical and numerical tools for solving these problems, c) to give you some hints as to how to interpret your results, which are often solutions of simplified versions of the real problems, d) to provide you with opportunities to become more comfortable with the use of computers for solving problems, and e) to demonstrate the potential value and limitations of these techniques in engineering practice.
6 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Design of A Bridge Over A Gorge
We want to build a bridge to span a gorge. Gorge Width
High Water
Gorge Depth
Low Water
How would we know a "good" bridge design? How would we know the "best" bridge design? By what process could we select a good design, or "the best" design?
Let's formulate the problem into objectives, decisions, and constraints.
What objectives could we have for the design? How can we quantify these objectives?
1) minimize cost ($) 2) maximize traffic volume (capacity in vehicles/hour) 3) maximize reliability of bridge/ minimize the probability of structural failure (failure due to earthquakes, floods, bombs, etc.) 4) make it pretty (hard to quantify, but often very important) 5) make it not cause flooding upstream (backwater effect) (Expected upstream flood damage) 6) Make it easy and safe to build. Who might choose some of these more specific objectives?
Decisions: What types of design decisions can the engineer make? 1) Geometry of bridge 2) Material of bridge 3) Size/capacity of bridge
Constraints: What factors constrain the design of any bridge over the gorge?
-) Geometry of gorge (distance to be spanned and depth of gorge) -) Geology of gorge (ability to support bridge; earthquakes) -) Hydrology and hydraulics of gorge (floods, scour, etc.) -) Structural constraints (properties of materials, conservation of momentum, statics, dynamics) -) Budget?
These constraints and objectives are all very complex. How could we ever hope to write complete mathematical descriptions of it all? (We can't.)
7 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Problem simplification Since we cannot solve the real problem in all its complexity, we'll have to simplify. Some ideas for simplification:
1) Reducing the number of objectives: Can some objectives be formulated as constraints? Are some objectives accommodated by any design or relatively unimportant, and can therefore be left out? 1) Bridge must cost < some budget amount. 2) Traffic capacity > some level of traffic demand. 3) Probability of structural failure < some annual reliability level. 4) It's cheap to make it pretty so we won't worry about this now. 5) If nothing valuable is upstream, any backwater is unimportant. If something very valuable is upstream, allow no backwater.
2) Simplifying the Constraints -) The different types of bridge geometries would be a mathematical nightmare. Let's look at only a few types of bridge that are commonly successful for such gorges, optimize the design of each type for this gorge, and then compare the "best" of each bridge type to select the one bridge we'll build.
-) Can the mathematics of any constraints be approximated by a simple mathematical equation or inequality?
Formulation for a Simple Bridge Type: A simple geometry of spans supported by pilings. Spans
Pilings
Simplified Decisions: How many pairs of pilings and how many spans should be used?
Simplified Objective: Minimize total construction cost, CT 1) Cost of a pair of pilings = a given function of compressive force, fp(F)
Cp = fp(F), F = decision of compressive force on a single piling pair
2) Cost of span = a given function of its length, fs (L):
Cs = fs (L), L = decision of length of each span.
3) If N pairs of pilings are used, N+1 spans will be needed. Therefore, total cost is approximated by: CT = N Cp + (N+1) Cs, N = decision of number of piling pairs
8 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017
4) Static design: The load on each piling pair is the weight of a span and the live load (vehicles). This load is a function of the span's length, where we'll assume w is a constant.
F = wL
5) The width of the gorge must be spanned
(N + 1) L = g, g = gorge width (another constant) g or L = N + 1
Only one "decision variable" remains!
Now, Minimize Cost for either L, F, or N and the other 2 values are known
Reduce the known objective function from Step 3 to one independent decision variable. CT = N Cp + (N+1) Cs goes to: CT = N fp(F) + (N+1) fs(L) goes to: CT = N fp(wL) + (N+1) fs(L) which finally goes to: w g g C = N f + (N+1) f T pN + 1 sN + 1
where the entire cost function is in terms only of the number of piling pairs N. What's the optimal N?
Solution Methods 1) Trial and Error: Try different values of N and see which gives the lowest cost. 2) Enumeration: Try all or many values of N and see which gives the lowest cost. 3) Other more sophisticated methods (we'll learn later).
Let's solve by enumeration. Solve the boxed equation for all reasonable values of N, and plot the results:
Total Cost ($millions)
1 2 3 4 5 6 7 8 9 10 Number of Spans (N+1)
This is a simple problem, but it illustrates the approach. The course presents a mathematical theory of optimal design. Even if the details of the theory will sometimes be hard to implement, the approach should lead to better, if not "the best" designs. For some this will represent a new way of thinking about engineering problems. 9 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Systems Analysis
Last time I tried to give a flavor of what is meant by "deterministic optimization."
Things I'd like to do today:
1) Place this course in the larger context of Systems Analysis and Operations Research 2) Discuss the role of Operations Research in civil engineering: real cases where it's used. 3) Suggest a general approach to applying optimization to civil engineering problems. To codify what we did with the example last time and illustrate with an example.
Systems Analysis/Operations Research
Broadly, the use of mathematical descriptions of problems to help find better solutions.
Formal origins: World War II, logistics and combat tactics problems.
Utility of computers! (to test many solutions quickly and to implement tedious algorithms)
Current applications in business (MBAs), military, engineering.
EXCEL spreadsheets include optimization “Solve” with some elementary linear programing, nonlinear-programming, and evolutionary algorithm capabilities.
Systems Analysis is more than just a bag of tricks and tools!
Systems and/Operations Research as a problem-solving discipline! A way of thinking about problems!
Optimization by any other Name, ...:
Optimization goes under a number of different names and constitutes an important part of many fields. Here are some:
Systems Analysis Operations Research (British: Operational Research) Industrial Engineering Management Science Engineering Economics.
I tend to use the term systems analysis, since it is fairly broad and to me it connotes a systematic analysis of design decisions.
Here is a nice talk by David Rosenberg on optimization and the value of near-optimization. https://www.youtube.com/watch?v=b96LsqcJyHM
10 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Some Uses of Systems Analysis in Civil Engineering (skipped in lecture) Water Resources
Operating reservoirs, pipelines, and pumping plants (MWD of So. Cal., PG&E, Army COE) Determining optimal maintenance schedules (dredging, water meters, etc.) Optimal pumping schedules (groundwater and surface water) Scheduling of water conservation and supply facilities (San Diego County Water Authority) Optimal use of water purchases (San Diego County Water Authority) Most efficient hydraulic cross-section (textbook hydraulics)
Environmental Engineering:
Least-cost treatment levels for wastewater plants on a river subject to water quality constraints Least-cost sequencing of wastewater treatment processes Least-cost planning of solid waste recycling Locating transfer stations for solid waste collection Determining collection routes for garbage trucks Selecting a most profitable mix of recycled products Least-cost design of material recovery facilities
Structures:
Designing structures that accommodate loads with minimum weight Design of least-cost or minimum-weight structural cross-sections
Geotechnical:
Least-cost selection of aggregate sources for an aggregate mix Least-cost disposal of dredged material
Transportation:
Assigning aircraft to scheduled routes Assigning railroad cars to routes Selecting cargos to fit on a ship Traffic flow optimization and signal timing Routing ships, aircraft, trucks, and rail cars
Construction:
Selecting least-cost or completion-time-minimizing schedule and sequence of construction tasks to finish a project (Critical Path Method - CPM) Least-cost cut and fill Construction equipment allocation between projects
11 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017
THE MATHEMATICAL OPTIMIZATION PROCESS
1) Define the Problem Problem Formulation 2) Express the Problem Mathematically 3) Simplify the Mathematical Problem to a Solvable Form
Problem 4) Solve the Simplified Mathematical Problem Solution 5) Sensitivity Analysis
6) Interpretation of Results Interpretation and Conclusions 7) Recommendations for Further Study or Implementation
12 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Steps to Solving a Problem Using Optimization
Here is a step-by-step approach to solving design problems with optimization. Hopefully, you'll find it useful in approaching some of your homework problems and projects.
1) Define the Problem (in words) a) What decisions are to be made? b) What is an objective(s)? c) What are the constraints?
2) Express the Problem Mathematically (Simplification Step 1) a) Identify decision variables. b) Define objective function(s). c) Express constraints mathematically.
3) Simplify Mathematical Problem Statement to a Solvable Form (Simplification Step 2).
Some Techniques: a) Express some objectives as constraints. b) Eliminate "minor" aspects of the problem. c) Adopt approximate expressions for objectives and constraints. d) Substitute equality constraints into the objective function.
4) Solve the Simplified Mathematical Problem Statement (using various methods presented in the course) a) Analytical solution methods (Calculus) b) Search Methods c) Mixed Analytical and Search Methods (Efficient Search Methods)
5) Sensitivity Analysis No solution is perfect. Simplifications and assumptions were made. -) How stable is the solution if problematic assumptions are varied? -) Effects of uncertainties on the optimal solution and system performance?
6) Interpretation of Results This is what people pay for. The challenging part of this step is to reveal the design implications of your solution, given that you have had to simplify the real problem before you could solve it. The solution to the math problem is not always the solution to the real problem. Sensitivity analysis and bit of thought can help a lot here.
7) More Analysis and/or Recommendations for Implementation? Is more analysis needed? What recommendations would you make for implementing your conclusions? Are there particular aspects of implementing the design that merit more attention?
13 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017
Example: Task Scheduling (Critical Path Method - CPM), 5-tasks in project.
Background: Task Duration (weeks) Pre-requisite Activities 1) Excavation 2 None 2) Building Construction 10 1 3) Internal Finishing 4 2 4) Parking Lot 1 1 5) Landscaping 3 2,4
Step 1: Define Problem
a) Decisions: At what time should each task be begun? (Schedule tasks) b) Objective: Min. time-to-completion of project. c) Constraints: Some tasks are pre-requisites of other tasks. Tasks also have durations.
Step 2: Mathematical Statement
a) Let Si = starting week of task i. i = index referring to activity Si are decision variables b) Min T = date of completion c) Constraints: S2 S1 + 2 T S1 + 2 S3 S2 + 10 T S2 + 10 S4 S1 + 2 T S3 + 4 S5 S2 + 10 T S4 + 1 S5 S4 + 1 T S5 + 3
Pre-requisites Definition of completion date (Completion occurs after every task is done)
Step 3: Simplification - No more simplification is needed.
Step 4: Solve - Today we will solve by common sense 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Excav. (1) Building Construction(2) Internal Finish(3) " Critical path" Parking Lot (4) Slack Landscaping(5) Slack
discuss
(Scheduling tens or hundreds of tasks is more easily done by computer.)
14 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Step 5: Sensitivity Analysis
The durations of each activity are always somewhat uncertain. Maybe Fred's bulldozer breaks down and takes an additional week to fix.
Task Change in Tmin with small change in task duration 1 1 2 1 These activities are the "critical path" 3 1 4 0 5 0
Step 6: Interpretation
1) Do activities in the order 1, 2, and 3. Schedule activity 4 for anytime between weeks 2 and 11. Schedule activity 5 for either week 12 or 13. 2) Any "rush" of the project should concentrate on Tasks 1, 2, and 3. 3) No need to hurry on the parking lot. 4) Activity 5 should be begun within a week of the beginning of activity 3, or the completion of the entire project will be delayed. 5) No need to hurry on landscaping unless Task 3 is accelerated by more than a week.
15 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Unconstrained Optimization Using Calculus Now we'll start with the presentation of algorithms for solving mathematical statements of deterministic optimization problems. Let's begin with some definitions that will be used all quarter.
Maxima and Minima Definitions Global Maximum Z = f(X) Local Maximum Z = objective value X = decision variable
Local Max Z
Local Min X
Local Minimum Global Minimum
-) Local versus global optima -) Maximum versus minimum
Note: A local maximum can be less than a local minimum. It is typically necessary to look at all "extreme" points (maxima or minima) to find the global optimum.
Differentiability When solving an optimization problem by calculus, the objective function must be "piece-wise" continuous, and twice differentiable;
df d2f and must exist over the range of x. dx dx2
Finding Unconstrained Maxima and Minima For Objective Functions of One Decision Variable
Let xo be some particular value of x that we want to examine for optimality.
First-Order Condition: (Necessary Condition) dz 0 , evaluated at xo dx x 0 Several values of x might satisfy this condition. Points satisfying this condition might be a maximum or a minimum. Points satisfying this condition are called critical points. Extreme points can also exist at discontinuities in the objective function and boundaries of the feasible region.
16 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017
Second-Order Condition: d2z If, in addition to the first-order condition, 0 , the point xo is a maximum. 2 dx x 0
The change in dz/dx is negative at x. Therefore, z must decrease with further increase in x, making the point a maximum.
Necessary and Sufficient Conditions for Xo to be a Maximum 1) dz = 0 dx xo and 2 2) dz < 0 dx2 xo
Necessary and Sufficient Conditions forX o to be a Minimum 1) dz = 0 dx xo d2 2) z > 0 dx2 xo
d2z If, in addition to the first-order condition, = 0, the point xo is on a plateau, with several equally- dx2 optimal values of x.
How does a blind man find the top or the bottom of a hill?
17 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Finding Unconstrained Optima for Objective Functions of Several Decision Variables
Now, let the objective function be a function of a vector of decision variables: z = f x , x = x1, x2, - - -, xn , a vector of n decision variable
First-Order Condition: (Necessary Condition)
f x o
x1 f x o f xo = = 0 x2 f x o
xn
This is the same condition as before. The partial derivative of the objective function with respect to each decision variable must be zero. This is a necessary condition for a hill-top or a hole, no matter how many dimensions the geometry has.
Just using these first-order conditions will greatly reduce the number of potentially optimal decisions that need to be evaluated.
Second-Order Condition: The second-order conditions look a bit stranger, however.
For a minimum:
Hessian Matrix
T o Xn H ()Xi Xn 0 for all Xn 0 If this inequality holds, H is called T a "Positive Definite" or “Positive Xn = arbitrary vector with n elements Semi-definite” matrix
This is how the Hessian matrix is defined:
18 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017
2 2 2 f ( xo) f( xo) ff( xo) - - - 2 x1 xxf1 f2 x1 xn
| | | | H( xo) = | | | | | |
2 2 2 f( xo) f ( xo) f( xo) - - - 2 xn x1 xn x2 xn
T X H X > 0 H positive definite if n n .
f(X)
• X2 X 1
T H positive semidefinite if Xn HXn 0.
This is sometimes called a saddle point. f(X)
•
X 2
X 1
19 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Example 2 2 MIN f X = X1-1 + X2 + 1
o f X
0 X1 2 X -1 f X = = = 1 o 2X 0 f X 2
X2 X1 = 1, X2 = 0. 2 o 2 o f X f X
2 o X X1X 2 H X 1 = 20 2 o 2 o f X f X 02
X X 2 2 1 X2
Is H positive definite? T Xi H Xi 0 for all X 0
20 X1 ? X1,X2 02 > 0 X2
X1 ? 2X1,2X2 > 0 X2
2 2 2X1 + 2X2 >0, for any X 0
H is positive definite.
Second-Order Condition for a Maximum: X0 represents a maximum if f( X0 ) = 0 and HX0 is "negative definite" or "negative semi- definite"
H( X0 ) is "negative definite" if X T H X < 0 for all X 0
(What a mess to calculate!) (But these are rigorous rules!)
Note: X0 can satisfy f( X0 ) = 0 without being a minimum or a maximum, by being a "saddle point," or an unstable minimum-maximum. (i.e., min w.r.t. X1 & max w.r.t. X2).
We won't use Hessians hardly at all.
20 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Analytical vs. Search methods
We have already seen some limitations to completely analytical methods:
1) often time-consuming to analyze 2) does not distinguish between global and local optima
Still, analytical methods can sometimes easily solve problems, give more insight into the behavior of the optimal solution, and help simplify the solution of the problem to speed numerical searches.
Example: Optimal Container Dimensions
-) A cylinder must contain a given volume of oil, V. -) Find the height and radius of the cylinder that will minimize the cylinder's surface area.
Min S = 2rh + 2r2 V S.T. V = r2h h = r2
Min S = 2r V + 2r2 r2 or
2V 2 Min S = r + 2r .
1st-order condition:
dS -2V = 0 0 = + 4r dr r2
3 0 = 4r3 - 2V V = r* 2
2nd-order condition:
d2S 4V V = + 4 > 0 > - dr2 r3 r3
V If V = 10ft3, r* = 1.167 ft from 1st-order condition, and > - , satisfying 2nd-order condition for r3 a minimum of S.
21 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Example 2: Optimal Scheduling of Water Meter Maintenance
A water meter is used to charge a large industrial customer for water use. The water utility must decide how often to repair the meter, given that the accuracy deteriorates with time. The meter always deteriorates such that it registers less than the actual flow amount.
The average yearly volume used is 10,000 x 100 cu.ft. (ccf). The price of water is 50¢/ccf. The cost of repairing the meter (which restores accuracy) is $500. The meter loses 1% of its remaining accuracy each year.
Find how often the utility should repair the meter to minimize present value of total repair and revenue- loss costs. The interest rate is 5%.
The present value of costs and losses over one period between repairs of length T is
T C(T) = 500 + 0.5 1 0,000 ()1 - e-0.01t e-0.05t dt 0 1 1 = 500 + 5,000 1 - e-0.05T - 1 - e-0.06T 0.05 ()0.06 ()
e-0.05T e-0.06T C(T) = 500 + 5,000 3.333 - 0.05 + 0.06
An infinite series of these repairs, occurring at intervals T, has a present value cost of: PV(T) = C(T )e0.05(kT ) k0
Cost
0 T 2T 3T 4T … time
This infinite series simplifies to PV(T) = C(T) + PV(T)e-0.05T, which further simplifies to:
C(T) PV(T) = 1 - e-0.05 T
The objective is to minimize this present value cost.
C(T) Min PV(T) = 1 - e-0.05 T
-0.06T 500 + 100,000 - 83,333 1-e PV(T) = 1-e-0.05T 1-e-0.05T
22 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Solution by search/enumeration: PV Plot and select T.* T* T Change-out Period Present Value Cost A B 1 10742.8 2 6229.3 3 5036.6 3.5 4796.0 4 4665.69 4.5* 4614.94* 5 4616.98 5.5 4656.7 6 4723.9 6.5 4811.9 7 4915.4 8 5155.8 9 5425.5 10 5712.6 11 6009.8 12 6312.3
Solution by Calculus:
dPV(T) 1st order condition: Solve: dT = 0
0.05 500 0.01 e-0.06T -e-0.01T = - 0.05 0.06 5,000 0.06
0.167 e-0.06T -e-0.01T =-0.8283
Here, the first-order condition needs a numerical solution. T LHS 10 -0.8132 9 -0.8166 7 -0.8227 5 -0.8275 4.5 -0.8285 T* 4.5 years 4 -0.8294
23 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Lagrange Multipliers
A method of solving simple constrained optimization problems of the form:
Max or Min f(X1, X2, ..., Xn)
S.T. g1(X1, ..., Xn) = b1 g2(X1, ..., Xn) = b2 • • • gm(X1, ..., Xn) = bm
Example: Max z = X1 + X2 2 2 S.T. 2 X1 + X2 = 1
One way to solve these problems is to use substitution to incorporate constraints directly into the objective function. This is not always possible. Also, substitution will not work for inequality constraints.
Solving problems with a single constraint Max or Min Z = f X , ..., X Step 1: State Problem: 1 n S.T. g1 (X 1, ..., Xn) = b Step 2: Form the "Lagrangian":
L f (X 1 ,..., X n ) (b g1 (X 1 ,..., X n )) "Lagrange Multiplier", Treat like just another variable.
Step 3: Treat the Lagrangian function like any other function to be maximized or minimized, adding as another decision variable.
a) Solve for the first-order conditions: L = 0, Vi Xi L = 0 (returns constraint)
b) Determine if solutions to first-order conditions are maxima or minima. -) Using Hessian matrices or evaluation of the objective function at each point. -) Select "best" solution
Step 4: Interpret , the Lagrange Multiplier
f max or min = b
is the change in the best value of the objective function, if the constraint b is changed. 24 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017
A simple example:
Step 1 Max f = X1 + X2 2 2 S.T. 2X1 + X2 = 1
Step 2
2 2 L = X1 + X2 + (1 - 2X1 - X2 )
Step 3
L = 1 - 4X1 = 0 (1) X1
L = 1 - 2X2 = 0 ( (2) X2
L 2 2 = 1 - 2X1 -X2 = 0 (3)
(1) = 1 4X1
(2) = 1 2X2
(1) & (2) 4X1 = 2X2 2X1 =X2 26 2 2 1 1 2 3 6 into (3) 1 =2X + 2/xX1 X1 = f = + = 1 6 6 6 6 2 2 3 X2 = = 6 2 6
Step 4: f = max = 3 26 b
fmax = b, for small b. 3 Raising b from 1 to 2 should raise f by 2 6. max
is called a Lagrange multiplier, shadow price, or dual variable.
25 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Solving Problems with Several Equality Constraints
Step 1 Max or Min f(X1, X2, ..., Xn)
S.T. g1(X1, ..., Xn) = b1 g2(X1, ..., Xn) = b2 • • • gm(X1, ..., Xn) = bm Step 2 Form Lagrangian
L = f X1,..., Xn +1 b1 - g1 X1,..., Xn
+2 b2 - g2 X1 ,..., X n • • • +m bm - gm X1,..., Xn
Step 3 Solve for Xi and j
L = 0, i, there are n of these X i L = 0, j, there are m of these j
This implies simultaneous solution of (n + m) equations. So this method is usually used on only small problems.
same interpretation as before fmaximum Step 4: j = b j
For the skeptical reader, a derivation of the Lagrange multiplier method appears in an appendix.
26 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Inequality Constraints
Inequality constraints are very common.
Given an inequality constraint; g(X1,..., Xn) b:
The constraint is called non-binding if g(X*, …, Xn*) > b
The constraint is binding if g(X*, …, Xn*) = b
For a problem:
Max f(X1, X2) S.T. g1(X1, X2) b1 g2(X1, X2) b2
L = f(X1,X2) + 1(b1-g1( X )) + 2(b2-g2( X ))
If constraint 1 is non-binding, 1 = 0, then no small change in b1 will affect the "best" decision, removing the constraint from the Lagrangian.
If constraint 2 is binding, 2 0, and the constraint remains in the Lagrangian.
Making inequality into equality constraints: gj(X1,..., Xn) bj becomes gj (X1,..., Xn) + Sj = bj, where Sj is a slack variable.
S = 0 if constraint is binding. j If Sj0, j=0. "Complementary Slackness" Sj j = 0 . We'll use this technique later as well.
Example: Multiple inequality constraints Step 1: Max Z = X + X 1 2 X1 2 2 S.T. X1 + X2 25 X1 4
Step 2: Form the Lagrangian. 2 2 X = 4 L = X1 + X2 + 1 (25 - X1 - X2 ) + 2 (4 - X1) 1
Step 3: Solve the Lagrangian for the first-order X2 conditions, both for the Xis and the js.
L Solve: (1) = 0 = 1 - 2 1X1 - 2 X 1 X 2 + X 2 = 25 1 2 L (2) = 0 = 1 - 2 1X2 X 2
27 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017
L (3) = 0, and May go away, depending on value assumed 1 L for 1 and 2 (4) = 0. 2
FOUR cases exist for the possible sets of Lagrange Multiplier values. For n inequality constraints, there are 2n cases.
1) 1 = 0 , 2 = 0 Solution: Z , X1 , X2 , but violates both constraints.
2) 1 = 0, 2 0: Solution: Z , X1 = 4, X2 , but violates 1st constraint.
3) 1 0, 2 = 0: 1st constraint binds; 2nd constraint is non-binding. There are two solutions for this case: 25 Solution 1: Z = 7.07, X1 = X2 = 2 3.54. OK with both constraints; 25 Solution 2: Z = -7.07, X1 = X2 = - 2 . OK with both constraints, but solution is a minimum.
4) 1 0, 2 0: Both constraints bind, OK with both constraints; Two solutions: Solution 1: Z = 7, X1 = 4, X2 = 3, Solution 2: Z = 1; X1 = 4, X2 = -3.
FOR EACH CASE: 1) Solve for values of the decision variables which satisfy the first-order conditions. 2) Eliminate solutions that violate the constraints. * * 3) Select from remaining solutions, the solution X1 , X2 , with the best value of Z. 4) From all the cases, select the solution with the best value of Z.
For many problems there are often several points that satisfy the 1st-order conditions of the Lagrangian, especially when there are inequality constraints.
To solve these problems, solutions for the 1st-order conditions are examined systematically, by a search procedure to find the "best" solution.
28 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Example
Step 1 Max Z = 5X1 + 3X2 + 6X1X2
ST 1) X1 + X2 6 2) X1 7 3) X2 3 4) X1 0 Step 2 L = 5X1 + 3X2 + 6X1 X2 + 1 (6 - X1 - X2) + 2 (7 - X1) + 3 (3 - X2) + 4 (0 - X1)
Step 3: For each combination of binding and non-binding constraints. (24 or 16 cases for this example) 1) Solve for X1 and X2 0 0 2) Eliminate solution if a solution X1 , X2 violates a constraint that was assumed to be non-binding.
Select the optimal solution from those remaining. (Optimal solution has best objective function value.) L L = 0, = 0, if j 0. X Feasible i j X 2 region L = 0 = 5 + 6 X2 – 1 – 2– 4 X 1 6
L (4) = 0 = 3 + 6 X1 – 1 – 3 X 2 (1) (2) 3 L (3) = 0 = 6 - X1 - X2, if 1 0 1
L = 0 = 7 - X1, if 2 0 2 0 6 7 X L 1 = 0 = 3 - X2, if 2 0 3
L = 0 = X1, if 4 0 4 Cases: At least 8 different solutions: 1) 1 0, 2 0, 3 0 2) 1 0, 2 0, 3 = 0 3) 1 0, 2 = 0, 3 = 0 4) 1 0, 2 = 0, 3 0 5) 1 = 0, 2 = 0, 3 0 6) 1 = 0, 2 0, 3 0 7) 1 = 0, 2 0, 3 = 0 8) 1 = 0, 2 = 0, 3 = 0 m There are 16 cases if we consider 4 too. For a system of m inequality constraints, there are 2 cases. (You can see how this can get ugly.)
Case 8: No constraints bind 29 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Solution A: L = 0 X = -5 2 6 X1
L violates constraints 3 and 4 = 0 X = -3 = -1 1 6 2 X2 Z = -2.5 + -2.5 + 5/22.5 2.5 = -2.5 - -2.5 a saddle-point solution
Solution B: not found using derivatives
violates constraints 1 and 2 X1 = X2 = The solution if no constraints were to bind. Z =
Solutions C & D:
X1 = - X1 = + violates constraints 1 and 2 X2 = X2 = - Z = -
Solution E: violates constraints 3 and 4 X1 = X2 = - Z = -
If we go along this route, for all the cases, we might finish this example by the end of the course. So, we'll have to think a bit to shorten this process. Note: If X2 = 0, then 0 X1 6, using all constraints. And, if X1 = 0, then 3 X2 6, using all constraints.
Therefore, Constraint 2 will never be binding. 2 = 0
Therefore, we only need to worry about cases where 2 = 0. Consider only cases 3, 4, 5, and 8. Case 8 is already eliminated since all of its solutions violate constraints.
Consider only cases: 3, 4, and 5.
Now, look at feasible region, bounded by constraints.
Probably 1 0 and (40 or 30). To shorten the search process.
L = 0 = 5 + 6X2 – 1- 4 X 1
30 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 L = 0 = 3 + 6X1 + 1- 3 X 2
If 10: X1 = 6 - X2 30: X2 = 3 X40: X1 = 0
If 10 & 30: X2 = 3, X1 = 3 Z = 60 10 & 40: X2 = 6, X1 = 0 Z = 18
If 10 and 3 = 4 = 0: By substituting binding constraint 1 into objective function: Z = 5(6 - X2) + 3X2 + 6(6 - X2) X2 2 = 30 - 5X2 + 3X2 + 36X2 = 6X2 2 = 30 + 34X2 - 6X2 Z 34 = 0 = 34 - 12 X2 X2 = = 2.83 which violatesX2 3. X 12 2 This means there is no hill or valley for the plot of Z(X2) within the feasible range of X2. Let's plot this and see what it looks like.
Z
3 X2 6 There is no max or min along the segment, except at the ends. Therefore, Z*max = 60, at X1* = 3, X2* = 3.
Alternative Method for Inequality Constraints Add (for ) or subtract (for ) a squared slack variable, and treat this slack variable like any other in the problem. (Reference?)
Advantage: Avoids considering multiple cases. Disadvantages: More variables to solve for; the new variables are non-linear.
Example: Step 1: Max Z = X1 + X2
S.T. X12 + X22 + X32 = 25 X1 + X42 = 4
31 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017
Now, instead of 4 unknowns, we have 6 unknowns (4 Xis and 2 js). This isn’t appealing if we have to solve by hand, but it’s not a problem if we let a computer do the work.
32 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Mathematical Programming
Our solutions of constrained optimization problems fall in the field of Mathematical Programming.
A mathematical program is any method used to solve a problem of the form:
Max or Min fX any number of equality & inequality constrain S.T. g X b
A linear program solves problems of the form:
Min or Max cT x s.t. Ax b x 0 -) Linear objective function: f(X) = c1X1 + c2X2 + ... -) Linear constraints: ai1X1 + ai2X2 + ... < bi , i = 1, …, m -) There are linear programming methods that quickly and easily solve such problems.
Other mathematical programs are called non-linear programming problems.
Most texts study Lagrange multipliers as a non-linear program solution method.
In fact, Lagrange multipliers are at the heart of the method for solving linear programs as well!
Some terminology:
-) A non-binding constraint is also called an inactive or slack constraint.
-) A binding constraint is also called an active or tight constraint.
f -) The Lagrange multiplier is also known as a shadow price. = best b
Lagrange multiplier, Constraint shadow price Binding, tight, 0 active Non-binding, slack, = 0 inactive
33 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Convex and Concave Functions (usually objective functions) Convex Concave Convex and Concave over entire range convex
f(X) f(X) • f(X) • • •
X X X
Convex or Concave over Certain Ranges
Neither Convex nor Concave as a Whole
Is the line connecting any 2 pts. on the function always above f(X) • • intermediate points on the function? • • If yes function is convex.
X
Convex and Concave Sets (usually constraint sets for feasible regions)
Convex Set Concave Set • X X 1 1 • • •
X X 2 2
Does the line connecting any 2 points in the set contained by the set? If yes, the set is convex.
For a convex constraint set:
If the objective function is convex, any minimum is the global minimum.
If the objective function is concave, any maximum is a global maximum.
34 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Unbound Maxima and Unbounded Minima
Let's look at: Max z = X2 versus Min z = X2
Z
o X
Min Z dZ = 0 = 2 X X = 0 dX
2 d Z = 2 > 0 X = 0 is a minimum. dX2
Max Z dZ = 0 = 2 X X = 0, But this is a minimum. dX
In the curve above, there are 2 maxima, X = + and X = - These are unbound maxima.
Unbound maxima or minima occur where the value of the objective function + (unbound maximum) or objective function value - (unbound minimum). dZ/dX = 0 exists only for finite minima or maxima. Having an unbound optimum solution usually indicates an error in the problem formulation.
Duality For every constrained mathematical program problem, there is a dual problem. They are re-statements of the same problem. The original problem is called the primal problem.
For a primal problem: Minimize f X S.T. g X b
The Wolfe dual problem is: Maximize L ( X , ) Subject to: L ( X , ) = 0 0 This is why is called a dual variable. In Linear programming the dual form is much simpler and more useful.
35 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Karush-Kuhn-Tucker (KKT) Conditions for an Optimal Solution
A series of conditions which must be met for a solution to a constrained optimization problem to be an optimum (max. or min.). For the Problem: Max or Min. f X S.T. g X b
with n decision variables and m constraints. m The Lagrangian is: L( X , ) = f( X ) + i [bi gi (X ] i1
And the KKT conditions are:
L (1) = 0, i = 1,..., n Gradient conditions) Xi
(2) gj X bj, j = 1,..., m Feasibility conditions (3) j b - gj X = 0, j = 1,..., m Complementary slackness (4)j 0, j = 1,..., m Non-negativity (5) X 0, i = 1,..., n Non-negativity i Points Xo satisfying these conditions are maxima or minima.
If, in addition to satisfying these conditions,
g X b is a convex set, and f X is a convex function, Xo is a global minimum
or
g X b is a convex set, and f X is a concave function, Xo is a global maximum
36 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Linear Programming
A linear program is a mathematical program where: 1) the objective function is linear and 2) all constraints are linear equalities or inequalities.
The standard linear program thus has the form:
Minimize Z = c1X1 + c2X2 + ... + cnXn (X1, ..., Xn)
Subject To: a11X1 + a12X2 + ... + a1nXn < b1 a21X1 + a22X2 + ... + a2nXn < b2 ... am1X1 + am2X2 + ... + amnXn < bm X1, X2, ..., Xn 0 where, aji = unit requirement of Xi involved in constraint j (all constants), bj = resource availability (RHS constant) for constraint j, ci = constant coefficient (unit value) of objective for Xi, Xi = a decision variable, and Z = value of the objective function.
This problem has n decision variables, m resource constraints, and n non-negativity constraints.
Note that aji, bj, and ci are all constants (data or parameters) in the model.
In matrix notation, this standard for becomes:
Max or Min c T X S.T. A X < b X 0
Is the objective function for a linear program convex or non-convex? Is the constraint set convex?
What does this imply for global optimality of a local optimum to a linear program?
37 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Today we have a graphical example of a product-mix type problem frequently solved by linear programs.
Clyde Motor Company Example A company has 3 factories that together produce two products: cars and small trucks.
Each factory specializes in certain components required for both products. Plant 1 makes truck bodies. Plant 2 makes car bodies. Plant 3 makes shared components and assembles trucks and cars.
Each plant has a limited capacity. Each product is sold for a fixed price with a fixed profit per unit.
Data: Product Capacity Available Plant 1 2 (units/hour) 1 1 0 4 2 0 2 12 3 3 2 18 Unit Profit $3 $5 --
Step 1: Define the Problem: -) Max profit -) S.T. limited plant capacities -) How many cars and trucks should be produced?
Steps 2 and 3: Express the Problem Mathematically:
Max Z = 3X1 + 5X2 = profit per hour of operation, say.
Subject To: (1) 1X1 4 (2) 2X2 12 (3) 3X1 + 2X2 18 (4) 1X1 0 (5) 1X2 0
X1 = Number of small trucks produced. X2 = Number of cars produced.
Explanation of constraints: (1) 1X1 4 capacity constraints (2) 2X 12 2 on production (3) 3X1 +2X2 18 (4) X1 0 cannot produce negative products (5) X2 0
38 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Step 4: Solve. A graphical solution: 1) Graph the constraint set and feasible region 2) Plot contour lines of the objective function
9 Z = 36 (1) 8 (3)
(2,6) Z = 30 7 (2) 6
5 Z = 15 4
3 (4) X2
2
1
(5) 0 1 2 3 4 5 6 7 8 9 10 X1
To solve, move contour lines out until they are no longer feasible. Note, Z > 36 is not feasible.
39 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Step 5: Sensitivity Analysis --- Graphical Sensitivity Analysis
Several pertinent questions:
1) How much could unit profits change and leave the optimal solution unaffected? (How stable is the optimal solution w.r.t. price fluctuations?).
2) How much would each of the non-binding constraints have to change before the solution would be changed? (What if a machine breakdown lowered the capacity of a plant?) (I.e., what is the value of the slack variable assoc. with each non-binding constraint?)
3) How much should we be willing to pay to lessen each of the binding constraints? (How much should we pay to improve capacity at Plants 2 and 3, which are operating at full capacity?)
4) What happens if plant production efficiencies change (or product input requirements change)?
Modify earlier graph to answer each question.
Binding constraints are: (2) capacity of Plant 2, to produce car bodies. (3) capacity of Plant 3, to cars and trucks.
Assumptions for LP:
1) Proportionality: -) Producing twice as much costs (profits) twice as much -) Producing twice as much uses twice as many inputs (plant capacity)
2) Additivity: All objectives and LHS of constraints can be expressed as sums or differences of linear terms.
3) Divisibility: Fractional decisions can be made. 2.34 doors can be produced. We will discuss relaxing this assumption in a few weeks.
4) Certainty: -) Rates of profit/unit, capacity req./unit, and capacity constraint levels are constant and well-known. -) All parameter values are constant and well-known.
Additional Terms: Feasible solution Feasible region Corner-point feasible solution
What’s the relationship between optimal solutions and corner-point feasible solutions?
40 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 The Simplex Method: A Graphical Interpretation
The simplex method is the most common method for solving linear programs.
It is an algorithm which efficiently searches through different combinations of binding and non-binding constraints.
The Convexity of Linear Constraint Sets:
Can you think of any set of linear constraints which is concave?
Plot:
X1
X 2
Any linear constraint set will be convex or infeasible.
Optimality on the Corner Add a linear objective function (Max or Min).
Given a linear objective function and linear constraints, the solution will be either 1) a corner point, or 2) between and including adjacent (and equally optimal) corner points.
The Simplex Method, Qualitatively
1) Initialization: Find a feasible corner-point to start with.
2) Iterative Step: Move to a better adjacent feasible corner-point solution.
3) Optimality test: STOP if no adjacent corner point solutions are better. Otherwise, repeat Step 2.
Easier said than programmed.
Run through an example. [Perhaps another extended example.]
41 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 The Simplex Tableau
Here's how computer solution methods store the Linear Program.
n For the problem: 0) Min Z = ci X i i1 n Subject to: 1) a1i X i b1 i1 n 2) a2i X i b2 i1 n 3) a3i X i b3 i1
The inequality constraints are first made into equality constraints by adding slack variables:
n 1) a1i X i S1 b1 i1 n 2) a2i X i S2 b2 i1 n 3) a3i X i S3 b3 i1
The "Initial Tableau" is then formed
Basic Eqn Coefficients of Variables Variable No. Z X1 X2 - - - Xn S1 S2 S3 RHS Z 0 1 c1 c2 - - - cn 0 0 0 0 (objective function) S1 1 0 a11 a12 - - - a1n 1 0 0 b1 S2 2 0 a21 a22 - - - a2n 0 1 0 b2 (constraints) S3 3 0 a31 a32 - - - a3n 0 0 1 b3
This is simply a way of stating:
Max or Min c T X + S S.T. A X + I S = b
The tableau is then manipulated using the simplex method and stores the intermediate and final solutions to the Linear Program. The manipulations are all linear algebra.
In this initial tableau, the slack variables are basic (non-zero) and each of the original decision variables is non-basic (equal to zero). Non-basic variables define the binding constraints which define a corner point being examined.
42 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 The Initial Tableau for the Clyde Motor Co. Example:
Coefficient of Variable Basic Eqn. Variable No. Z X1 X2 S1 S2 S3 RHS Z 0 1 -3 -5 0 0 0 0 S1 1 0 1 0 1 0 0 4 S2 2 0 0 2 0 1 0 12 S3 3 0 3 2 0 0 1 18
This is the standard form for a linear program: Problem is stated as a minimization problem. All constraints are equality constraints. All variables (decision and slack) are non-negative.
Basic and Non-Basic Variables All variables (decision and slack) are classified as either Basic or Non-Basic for each iteration of the Simplex algorithm. This classification of variables defines a particular feasible corner-point solution to the LP problem.
Basic Variables all have non-zero values and are associated with non-binding constraints. (Slacks have non-zero values) Only one non-zero coefficient appears in the tableau for a basic variable and that coefficient = 1. (Show examples for initial tableau.)
The objective function variable Z is always considered to be a basic variable.
Non-Basic Variables all have values of zero and are those associated with binding constraints. (Slack variables values = 0)
For a linear programming problem with n decision variables and m constraints, there will be m + 1 basic variables. Non-negativity constraints are not counted in the m constraints.
Why m+1 basic variables? The objective function (Z) is always basic; that's one. A corner in an n-dimensional decision-space (feasible region) will be bound in n-dimensions (for each decision variable dimension to be bound). The total number of constraints (including non-negativity) is n+m. n of these bind together to form a corner- point. Therefore, there are n non-basic variables representing n binding constraints. Thus m constraints are left non-binding and have slack or decision variables which are basic.
NOTE: There are also m + 1 rows in the simplex tableau.
Here's a simple algorithm for the simplex method. A one-page version of the simplex algorithm appears in an appendix.
43 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 The Simplex Method - An Algorithm
Step 1: Initialization: From the initial tableau.
This also represents the first trial solution, where all slack variables are basic. (Note that each slack variable has only one non-zero coefficient and that coefficient = 1.)
Optimality Test (Is trial solution optimal?) This current solution is optimal only if every coefficient in Eqn. 0 (objective function line) is non- negative ( 0). STOP if optimal. Why? If a coefficient in Eqn. 0 is zero, the associated variable is basic and zero. If a coefficient in Eqn. 0 is negative, the associated variable =zero and by increasing that variable (from zero) the value of the objective function can be lessened (improved).
If all coefficients in Eqn. 0 are 0, no variable can be changed to decrease the value of the objective function. This is then the/an optimal solution.
Step 2: Iterative Step:
Part 1: Find the new basic variable.
Rule: The new basic variable is the variable with the most negative coefficient in Eqn. 0.
This variable produces the largest decrease in the objective function per unit increase in value (from zero).
For the Clyde problem: X2 has the most negative coefficient in Equation 0 of the initial tableau and so is the new basic variable.
Since only m + 1 variables may be basic, one previously basic variable must become non-basic.
Part 2: Find the new non-basic variable.
The column of the new basic variable is called the pivot column.
Rule: For each row (excluding Equation 0 and any rows where the pivot column elements are negative) divide the RHS by the coefficient in the pivot column.
RHS Select the row with the minimum ratio of i . This is called the pivot row. Cpivot i . This pivot row constraint will bind first as the new basic variable increases from zero. This leads to a feasible adjacent corner. Rows with negative coefficients in the pivot column are excluded because this would imply decreasing the new basic variable to a negative, and therefore infeasible, value.
The new non-basic variable is the old basic variable in the Basic Variable column of this row. The variable has a coefficient of 1 in this row and a coefficient of zero in all other rows.
44 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 The ratios for the first iteration of the Clyde problem are:
Associated Basic Row/Eqn. X2 RHS /C Variable Number coefficient RHS i X2,i Z 0 -5 0 Not Applicable S1 1 0 4 4/0 = S2 2 2 12 12/2=6 Min Ratio: S2 is new non-basic variable S3 3 2 18 9
The tableau must now be changed to reflect this new corner-point feasible solution.
Part 3: Find the new tableau
Rule: Use linear algebra operations to modify the tableau-matrix so that the New Tableau has zero coefs. in the column of the new basic variable, except for the pivot row where the coefficient = 1. Adding and subtracting multiples of the pivot row to the other rows is usually the most effective approach.
For the Clyde example: 1) Update Row-Eqn. 3 by subtracting Row 2 from Row 3. This produces a zero in column X2 of Row 3. 2) Update Row 2 by dividing Row 2 by 2. This produces a 1 in column X2 of Row 2. 3) Row 1 already has a zero in column X2. 4) Add 5 times Row 2 to Row 0. This puts a zero in column X2 of Row 0.
It is an easier process if the new tableau is written directly beneath the old tableau. Add new tableau beneath old tableau. (Leave room for one more iteration).
Coefficients Basic Equation Variable Number Z X1 X2 S1 S2 S3 RHS Z 0 1 -3 -5 0 0 0 0 Iteration 0 S1 1 0 1 0 1 0 0 4 S2 2 0 0 2 0 1 0 12 S3 3 0 3 2 0 0 1 18 Z 0 1 -3 0 0 5/2 0 30 Iteration 1 S1 1 0 1 0 1 0 0 4 X2 2 0 0 1 0 1/2 0 6 S3 3 0 3 0 0 -1 1 6 Z 0 1 0 0 0 3/2 1 36 Iteration 2 S1 1 0 0 0 1 1/3 -1/3 2 X2 2 0 0 1 0 1/2 0 6 X1 3 0 1 0 0 -1/3 1/3 2 Talk about second iteration later.
45 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Note: For the new tableau the value of each -) non-basic variable = 0 -) basic variable = RHS.
Step 3: Optimality test (again).
Are all coefficients in Eqn. 0 are non-negative? In example, No. A new iteration is needed. Repeat Step 2. ______
These are the fundamental steps in the Simplex method. Now let’s finish the problem.
Step 2 Iteration Step
Part 1: X1 is the new basic variable (only variable with negative coefficient in Row 0).
Part 2: Basic Coef of RHS/ X1 Variable RHS Coefficient of X1 Z -3 30 Not Applicable S1 1 4 4/1=4 X2 0 6 6/0 = S3 3 6 6/3=2; Make S3 the new non-basic variable.
Part 3: Produce New Tableau 1) Add Row 3 to Row 0 to set coefficient for X1 = 0 in Row 0. 2) Divide Row 3 by 3 for the new Row 3. 3) Row 2 is OK as is. 4) Subtract the new Row 3 from Row 1 to set coefficient of X1 = 0 in new Row 1. Add changes to tableau from p. 8.
Step 3: Optimality Test All coefficients in Row 0 0, so no other adjacent solution is better. Therefore STOP.
The final solution: Read from Tableau No. 2.
From RHS Column: Z = 36 S1 = 2 X2 = 6 X1 = 2.
From Top Row (Equation 0): 1 = 0 2 = 3/2 3 = 1 A short one-page description of the simplex algorithm follows.
46 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 THE SIMPLEX ALGORITHM (in one page)
The following steps form the fundamentals of the Simplex Algorithm for solving linear programs.
1) Set up an "initial tableau."
2) Is the trial solution represented by the initial tableau optimal? If all coefficients in Row 0 are 0, then the solution is optimal and you may STOP. If any coefficients in Row 0 of the tableau are < 0, the solution represented by this tableau is not the optimal solution and the set of basic variables must be changed.
3) Select a new basic variable. Select as a new basic variable the variable with the most negative coefficient in Row 0. The column for this variable is the "pivot column."
4) Select a new non-basic variable. Divide the RHS entry of each row by the coefficient in the column of the new basic variable. The new non-basic variable is the old basic variable associated with the row having the minimum ratio. This row is called the "pivot row."
5) Update the tableau. Form a new tableau that has zero for each entry in the pivot column, except for the coefficient in the pivot row. The entry in both the pivot row and pivot column must be equal to one. Use linear algebra and Gaussian elimination techniques to do this, centered on the pivot row. Replace the old basic variable with the new basic variable in the "Basic Variable" column. This new tableau represents a new corner-point solution.
6) Is this new solution optimal? If all coefficients in Row 0 of the new tableau are 0, then STOP; the solution is optimal. If any coefficients in Row 0 of the tableau are < 0, the solution represented by this tableau is not the optimal solution and the set of basic variables must be changed. GO TO STEP 3.
Reading the final tableau. The following tableau represents an optimal solution and final tableau for the problem: MAX 2X1 + X2 Subject To: 1) X1 + X2 4 2) X2 3
Basic Eqn. Coefficient of Variable . Variable No. Z X1 X2 S1 S2 RHS Z 0 1 0 1 2 0 8 X1 1 0 1 1 1 0 4 S2 2 0 0 1 0 1 3
The values of each basic variable is given by the RHS column. The value of each non-basic variable is zero. Z = 8, X1 = 4, S2 = 3, X2 = 0, S1 = 0. The value of the Lagrange multiplier (shadow price) for each constraint is given by the coefficient in Row 0 under the associated slack variable (S1 or S2). 1 = 2; 2 = 0. The Reduced Cost of each coefficient in the objective function is given by the coefficient in Row 0 under the associated decision variable (X1 or X2). Can you obtain these results?
47 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Some Simplex Details
The basic simplex method described earlier applies only to a relatively narrow set of linear programming problems. In particular, the problem must be a MINIMIZATION problem and all constraints must be of the form aij Xi bj, i.e., all constraints must be less than or equal to constraints. There are also some ambiguities in the application of the simplex method. This lecture will try to clear up these problems.
Maximization The simplex method minimizes the value of an objective function. But note:
Max z = Min (-z)
Min z = Max (-z).
So we can always make a maximization problem into a minimization problem, or vice versa.
Ties for New Basic Variable If two or more variables have the same negative coefficient in Row 0 of the tableau, and that coefficient is the most negative coefficient in Row 0, then select any variable from those with the tie. The choice is arbitrary.
Ties for New Non-Basic Variable What if 2 or more variables tie on the minimum ratio test for becoming the new non-basic variable? Break the tie arbitrarily. Possibility of cycling problems (discussed in book). Very rare.
Unbound Objective Function (Max z or Min z ) An unbound solution occurs if the new non-basic variable has a RHS/pivot ratio = , i.e., if Min(RHS/pivot) = . For this case that value of the new basic variable is not limited by any constraint and becomes infinite.
Multiple Optima Where the optima fall on two or more corner-points, the optima will also include the edges of the feasible region between these points.
These cases will appear only when, after the STOPPING criterion has been reached, at least one non- basic variable has a coefficient of zero in Row 0. If this variable is increased (from zero) it will not affect the objective function. The other corner optimal corner point solutions are found by performing extra tableau iterations.
48 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Equality Constraints Equality constraints are of the form:
aij Xi = bj, bj 0.
The problem with equality constraints is that the initial solution, X = 0, is no longer feasible.
To fix this, we add a new type of slack variable, called an artificial variable. This type of slack variable acts as a regular slack variable in the simplex method. The new equality constraint becomes:
aij Xi + Rj = bj
We also add a huge penalty to the objective function if Rj > 0. The new objective function becomes:
Min z = ci Xi + M Rj.
M is a HUGE number compared to the other coefficients ci.
These changes force the simplex method to quickly move away from the initially infeasible initial solution to a real feasible corner point. This approach is called the BIG M method.
Inequality Constraints inequality constraints are of the form:
aij Xi bj, or, with a slack variable, aij Xi - Sj = bj.
The same problem arises with inequality constraints as with equality constraints, i.e., the initial tableau does not represent a feasible solution. Unless bj 0, X cannot be equal to zero and the initial trail solution is infeasible.
We solve this problem again using the Big M method. First, an artificial variable is added to the constraint:
aij Xi - Sj + Rj = bj.
Second, a large penalty is given for having Rj > 0 by adding a HUGE coefficient for Rj in the objective function:
Min z = ci Xi + M Rj., where M is very large.
This quickly drives Rj 0 after the initial tableau and points the simplex method towards the feasible region.
Essentially, the BIG M method allows infeasible solutions, but only at an overwhelming penalty.
49 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017 Negative RHS Sometimes constraints have the form:
aij Xi bj, where bj < 0.
This can cause the initial tableau solution (all X = 0) to be infeasible. (This problem persists if we multiply both sides by negative one and reverse the inequality and becomes similar to solving a inequality constraint.) We solve this again using the Big M method.
aij Xi + Sj - Rj = bj or - aij Xi - Sj + Rj = ABS(bj)
Min z = ci Xi + M Rj., where M is very large.
No Feasible Solution This case can only occur if there are equality constraints or inequality constraints in addition to regular constraints. Therefore, artificial variables must appear in the formulation.
No Feasible Solution Two Parallel Equality Constraints No Feasible Solution
This case is discovered if the solution to the problem has at least one artificial variable not equal to zero.
Allowing Negative Values for a Decision Variable How could we let a decision variable Xi have any value, including negative?
+ - + Must divide Xi into two new decision variables: X i and X i. X i will represent Xi if it has a positive value - and X i will represent negative values of Xi.
We then replace Xi in all constraints and the objective function with:
+ - Xi = X i - X i.
We can similarly add a lower bound to the negative value of Xi by adding another constraint, where Li is the absolute value of the lower negative bound:
- Xi Li. This ensures that Xi -Li.
50 Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017
Minimizing a fractional objective
Given the objective and constraints,
Min z = ∑ / ∑ Subject to: ∑ 0
Transform variables so: Yi = T Xi, and let T = 1/ ∑
Re-write the formulation in terms of Yi and T, and multiply all constraints by T (assuming T >0), so the formulation becomes:
Min z = ∑ Subject to: ∑