On the Development of Optimization Theory
Total Page:16
File Type:pdf, Size:1020Kb
The American Mathematical Monthly pp ON THE DEVELOPMENT OF OPTIMIZATION THEORY Andras Prekopa Intro duction Farkass famous pap er of b ecame a principal reference for linear inequalities after the publication of the pap er of Kuhn and Tucker Nonlinear Programming in In that pap er Farkass fundamental theorem on linear inequalities was used to derive necessary conditions for optimality for the nonlinear programming problem The results obtained led to a rapid development of nonlinear optimization theory The work of John containing similar but weaker results for optimality published in has b een generally known but it was not until a few years ago that Karushs work of b ecame widely known although essentially the same result was obtained by Kuhn and Tucker in In this pap er we call attention to some imp ortantwork done in the last century and b efore We show that fundamental ideas ab out the necessary optimality conditions for nonlinear optimization sub ject to inequality constraints can b e found in pap ers byFourier Cournot and Farkas as well as by Gauss Ostrogradsky and Hamel To start to describ e the early development of optimization theory it is very helpful to lo ok at the rst two sentences in Farkass pap er The natural and systematic treatment of analytical mechanics has to have as its background the inequalit y principle of virtual displacements rst formulated byFourier Andras Prekopa received his PhD in at the University of Budap est under the leadership of A Renyi He was Assistant Professor and later Asso ciate Professor of Probability Theory at the same university Since he has held a professorship of mathematics at the Technical University of Budap est He is also head of the Applied Mathematics Department at the Computer and Automation Institute of the Hungarian Academy of Sciences He is a corresp onding memb er of the Hungarian Academy of Sciences the National Academy of Engineering of Mexico memb er of the ISI and Fellow of the Econometric So ciety His main interests are in Probability Statistics and Op erations Research including applicati on in engineering design water resources electrical engineering economics and natural sciences This pap er was prepared while the author was visiting the Mathematics ResearchCenter of the Universityof Wisconsin Madi son Readers who are not familiar with the sub ject whose history is discussed in this article should consult Pourciaus article in this Monthly Editors and later by Gauss The p ossibilityofsuch a treatment requires however some knowledge of homogeneous linear inequalities that maybesaidtohave b een entirely missing up to now We see that Farkas had a denite reason for developing the theory of linear inequalities He was Professor of Theoretical Physics at the University of Kolozsvar We can certainly assume that he himself had already applied his inequality theorem to the problem of mechanical equilibrium In fact he rep orted rst on the applications of the mechanical principle of Fourier at the session of the Hungarian AcademyheldonDecemb er This was published later in Hungarian and in German We shall analyze this pap er in more detail in further sections We present here however a brief summary of the problem and the principal result The metho d of Lagrange for nding extrema of functions sub ject to equality constraints was published in in his famous b o ok Mecanique Analytique as a to ol for nding the stable equilibrium state of a mechanical system In the case when a p otential exists the Lagrangian problem can b e formulated in the following manner minimize g x sub ject to g x i m i where the ob jective function is the p otential The case of inequality constraints was rst investigated in byFourier If a p otential exists then Fouriers problem is minimize g x sub ject to g x i m i Assuming that the constraining and the ob jective functions are dierentiable and some regularity conditions hold the Lagrangian necessary condition for the equilibriu m states that at the minimizing p oint x the gradient rg x can b e expressed as a suitable linear combination of the gradients rg x i m i For Fouriers problem the necessary condition for equilibrium was proved byFarkas in and This condition formulated without pro of for sp ecial cases by Cournot in and for the general case by Ostrogradsky in is that at the minimizing p oint x the gradient rg x can b e expressed as a linear combination with nonnegative co ecients of those gradients rg x whose corresp onding constraints hold i with equalityatx Figure illustrates the situation The constraints restricting a mass p ointareve x plane coincides with the halfspaces Cho osing the co ordinate system so that the x plane of the ground the p otential corresp onding to the gravitational force is given bythe wellknown formula mg x provided x is small as compared to the radius of the earth At the equilibrium p oint x four constraints hold with equality and the cone generated by their negative gradients contains the vector of the gravitational force x ∗ Figure The mass p oint is in stable equilibrium at x the vector of the gravitational force is in the cone generated by the negatives of the gradients of the constraining functions at x Nowwe need to say a few words ab out those mechanical principles which played an imp ortant role in the development of optimization theory On the Principles of Mechanical Equilibr iu m The principle of virtual work was enunciated by Johann Bernoulli in It app eared rst in a b o ok byVarignon in the same year Let us quote Lagrange I pp The principle of virtual velo city can b e given the following general form Ifasystemofanynumb er of b o dies or p oints on eachofwhichany forces act is in equilibrium and one gives the system some small displacement on the basis of which eachpointmoves an innitesimal distance that will express its virtual velo city the sum of the forces eachmultiplied by the distance moved by the corresp onding p oint in the direction of the force will always b e equal to zero if innitesimal displacements in the direction of the forces are taken as p ositive and those in the opp osite direction negative John Bernoulli is the rst as far as I know who observed this great general form of the principle of virtual velo city and its usefulness for the solution of the problems of Statics One can see this in one of his Letters to Varignon dated whichVarignon placed at the b eginning of the ninth Section of his New Mechanics a Section whichis entirely devoted to showing by means of dierent applications the truth and the use of the principle in question This principle was considered by Lagrange as an axiom of Mechanics On page of the same work we read In this law lies what is generally called the principle of virtual velo city whichhas b een recognized for a long time as the fundamental principle of equilibrium as we have describ ed it in the preceding Section and we can consequentlyconsideritasa kind of an axiom of Mechanics For the case of a conservative system of forces ie when the forces are given as nega tive partial derivatives of a scalar function the principle rst enunciated by Courtivron applies Concerning this Lagrange writes the following I p which yields a further principle of Statics namely that out of all congurations that are successively assumed by the system the one in which it has the greatest or the smallest kinetic energy is also that in whichitwould have to b e placed in order for it to remain in equilibrium See Courtivron Memoires de lAcademie des Sciences of and Lagrange gave sucient conditions that the p otential takes its minimum As Bertrand remarked his pro of was incomplete and Dirichlet later gave a correct pro of The mechanical principle of Fourier was rst published in in his pap er Memoires sur la Statique This concerns the case of inequality constraints Fark as remarked p that the declaration of the inequality principle is not the main contribution of the pap er It has the subtitle Contenantlademonstration du princip e des vitesses virtuelles This very general pro of however has not b een accepted see eg thus the main merit of the pap er is still contrary ot Farkass remark the statementofthe inequality principle We quote from page of Fouriers pap er As it frequently happ ens that the p oints of the system are only constrained by xed obstacles without b eing attached to them it is evident that there are p ossible displacements that do not satisfy the equations of constraint one also sees that for these displacements the moment of the resulting forces is necessarily p ositive since the direction of these forces has to b e orthogonal to the resisting surfaces Thus the sum of the moments of the applied forces is p ositive for all displacements of this kind but it is imp ossible to displace a solid b o dy that is in equilibrium so that the total moment of the applied forces will b e negative Finally if one considers the resistances as forces whichprovide us as weknow with the means of estimating these resistances the b o dy can b e considered free and the sum of the moments is zero for all p ossible displacements The moment of the forces P Q R acting on a mechanical system is dened as the sum of scalar pro ducts Pp Q q R r where p q r are variations of the displacements The Bernoulli principle declares to b e equal to zero while the Fourier principle declares to b e less than or equal to zero in case of equilibrium Fourier was using the uxion as it turns out from other parts of his