DOCSLIB.ORG

Derivative Characterization of Constrained Extrema of Functionals

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Derivative Characterization of Constrained Extrema of Functionals DERIVATIVE CHARACTERIZATION OF CONSTRAINED EXTREMA OF FUNCTIONALS A SURVEY Mohamed El-Hodiri University of Kansas Research Paper No. 34 AUGUST 1970 RESEARCH PAPERS IN THEORETICAL AND APPLIED ECONOMICS DEPARTMENT OF ECONOMICS THE UNIVERSITY OF KANSAS LAURENCE, KANSAS INTRODUCTION One of the basic axioms of economic analysis is the axiom of ration-.: ality, idea,of postulating that economic behavior results from a process of optimization. As it is, at best, difficult to directly test the axiom, economists have been interested in characterization theorems of optirniza- tion theory. In this paper we review the theorems that characterize opti- mality by way of derivatives. First we formulate a very general optimiza- tion problem. Then we present characterization theorems for three types of problems: Finite dimensional, variational and problems in linear topo- logical spaces. In each case we present theorems for equality - inequal- ity constraints. The theorems in each case are: first order necessary conditions, first order sufficient conditions, second order necessary conditions and second or~er:,sufficienTy&osditions. The scheme of repre- sentation is as follows: Statements of theorems are followed by remarks. referring the reader to the earliest, known to us, proofs of the theorems. In some instances, slight generalizations of some theorems appear here for the first time, an indication of necessary modifications to existing proofs are indicated. A case is "solved" if proofs for all the four types of characterization theorems exist. The only "unsolved" case is that of problems in linear topological spaces with inequality and with equality - inequality constraints. For this case, we present two conjectures about second order conditions that are analcgousto the equality constraint case. 2 We now state the "general" optimization problem: Let A,B,C,D be real linear topological spaces. Consider the func- tions : Let P be a partial ordering of By let >- be defined on C - as usual - by way of a convex cone and let el e2 e3 and e4 be the neutral element of 33' addition - zero elements - of A,B,c and D respectively. The problem may (1) be stated as follows .- A a Find E A such that f (xf is P-maximal (2) subject to: In case B is the real line, P is the relation ">"- defined for real numbers and we have a problem of scaler optimization. In case B is finite dimen- sional we have..a finite-vector maximization problem. In general, charac- terizations of solutions to finite-vector maximization problems may be derived from characterizations of solutions of scaler maximization prob- lems. We shall restrict our presentation to scaler maximization prob3ems. As an application of scaler maximization theorems we may "solve" a part- ticular finite-vector maximization problem, namely for the case where P is taken to be a aret to'^) ordering of B. FQ' infinite-vector maximization problems, a scalerization theorem appears to be the most appropriate inter- 3 mediate step to derive characterization theorems. Such a theorem was proved by Hurwicz [20]. The method, for finite-vector m3ximization, con- sists(') of simply observing that the problem is equivalent to an in- finite number of scaler maximization problems. However, we shall only be concerned with scaler maximization problems. 2. FINITE DIMENSIONAL PROBLEMS In this part we let A = E", B = E', C = E~ and D = E', where E", E~,Em II. E Euclidian spaces of dimensions n, one, m and R respectively. For theis part we reformulate the scaler maximization problem as follows: Problem 1. Find x E E~ such that f (x) >- f for all satisfying B ga(x) = 0, a = 1,..., m, h (x) 20,- B =1,..., R . .- We further state some properties of the constraints that will be used in discussing the theorems in this section. Some of the nomenclature, designated by " " is Karush's C231. 0 D.1) Definition. Effective constraints. 'Let xbe a point that satisfies hB 0 B (x) >- 0, = 1,.. , m. Let r (R) be the set of indices B such that h ($1 = 0. The constraints with indices B E r i:) 'kill be referred to as the 0 effective constraint6 at x. D.2) Definition. "Admissible Diyectiop'! Let h be differentiable. We say that X is an admissible direction if X is a non-tridal. solution to the inequalities: 0 D.3) Definition. "A curve issuing from x in the direction X." By that we mean,. an n-vector valued function, §(t), of a real variable t such that d E(0) = $ and 5 '(0) = - = A'. dt S(t) Itzo 0 D.4) Definition. admissible arc issuing from x in the direction A" is 0 an arc issuing from x in the direction X such that h (S(t)) >- 0. 0 D.5) Definition. "Property Q" -for inequality constraints, is satisfied at x. iff: For each admissible direction A, there exists an admissible arc issuing from 8. D.6) Definition. The rank condition for inequality constraints. We say that the rank condition h is satisfied at $ iff 1.) is differentiable and 2) the B rank of the matrix cR. I, f3 E r ($) , equals the number of effective con- 1 straints. D.7) Definition. The rank condition for equality_- inequality constraints is -- - -- 0 satisfied at x iff 1) the functions g and h are differentiable and 2) the 0 ag" 9 constraints at x, where EOL.i = -ax. I x =x: L .> ,.: . ..) D.8) Definition. The rank condition for equality constraints is satis- fied at if the rank of the matrix is a. [E?I1 Finally wa define local and global constraint maxima: D.9) Definition. x is said to be a local solution to problem 1. iff: A ,. ,. There exists a neighborhood of x, N(,x), such that f(x) >- f(x) for all ,. ? r N(;) satisfying g(*) = 0 and h(x) 2 0. ,. ,. ,. D.lO) Definition. x is said to -.be a global solution to.problem 1. iff f(x) 3 f(x) for all x satisfying g(x) = 0 and h(x) > 0. 2. 1. First order necessary conditions. +. Theorem -1. If f, g -and h -are contjnuously differentiable and if x --is a global solution to problem 1, then there exists a vector (A0' v,p) = (Ao, 1 m L v v ,e ,pa) 0 such that, 1 > 0 and ; ,. , ,. 0 - 1, eB>0,- ,OhB (;) =o.. 2) Fx- 0 = 0, where F0 = X f + w. g + p. h, Fx0 is the vector of partial 0 A derivatives of F with respect to the component's of x evaluated at k = x. n For the case of equality constraints, i .e. the set {x I h (x) 0 )=E , (6) Theorem 1 was proved by ~aratheodor~(~)C93 and Bliss C71. For the case n of inequality constraints, i.e. the set {x I g (y) = 0) = E , theorem 1 - except for the non-negativity of p in condition 1 of the conclusion - was proved by ~arush'~)[231. The non-negativity of p may be proved by using a separation theorem (8 . This was, essentially, the crux of Fritz John's [22] proof who has treated a more general problem of an infinite number of inequality constraints. For the equality-inequality constraints the proof involves writing equality constraints as inequality constraints (g lo and - g (x) >- 0) and applying Karush's or Fritz Jone's theorem. Theorem 2. -If, in addition to the assumptions of theorem 1 we have either.(9) . a) Property Q for inequality constraints, D. 4, and the rank condition for .. .. equality constraints, D. 7, are satisfied at x. I a ) --The rank condition ~-for .. equality -inequality constraints is satisfied A -at x. Then the conclusions of theorem 1 follow with X 0. ---L--I)- For equality constraints the theorem was proved as a corollary of theorem 1 by ~liss(lO' [7]. For the case of inequality constraints the theorem was proved by Karush (I1) C231. The proof, for equality-inequality constraints, may be accomplished by converting inequality constraints to equality constraints and obtaining the theorem as a corollary to theorem 1. This was presented by Pennisi (13) C271. A direct proof was presenied by Hestenes (l" [l9]. 2.2. First Order Sufficient Conditions. Theorem 3. If: 1) f. E and h are differentiable 2) The conclusions of A ,. A theorem 1 hold with X_ > J at a point x with g(x) = 0, h (x) > 0 3) " . Either (3.a) FO, of theorem 1, is concave or (3.b) G! is linear and h is ,. concave. Then x is a rloh31 solution to ~roblem1. The theorem follows from the fact that a concave function lies above its tangent plane. The implications of this fact was utilized by Kuhn- Tucker C241 in the proof of their equivalence theorem. The present the- orem may be proved by applying Lemma 3 of Kuhn and Tucker C241 to FO. 2.3 Second Order Necessary Conditicns. Theorem 4. If: 1) --f, g and h have second order continuous partial de- A rivatives. 2: x is a solutLon-- to problem 1. 3) The rank condition for equality - inequality ccastraints , 9.5. I.... -- 5 .I--- . ..-!:. 1 m 1 R Thenthere exis-? multipliers ( v,p) = (v ,. :, v ; LI ,. , LI ) such that: For the equality constraints, thc! theorem was proved by Caratheodory (15) [9] and ~Liss(~~)['7].For inequality constraints the theorem was proved by Karush (17) [231. For the equality-inequality case, the theorem was (19) proved by Pennisi (lR) [27], under the, dispensible , stipulation that the number 02 no~l-~sromultipliers attached to effective inequality con- straints is at most one. 2.4. Second order sxffi,.ent conditions. Theorem 5,. If: l? f, g c7d h have continuoas second order derivatives.
Load more

Recommended publications
  • Some History of the Conjugate Gradient and Lanczos Algorithms: 1948-1976
    Some History of the Conjugate Gradient and Lanczos Algorithms: 1948-1976 Gene H. Golub; Dianne P. O'Leary SIAM Review, Vol. 31, No. 1. (Mar., 1989), pp. 50-102. Stable URL: http://links.jstor.org/sici?sici=0036-1445%28198903%2931%3A1%3C50%3ASHOTCG%3E2.0.CO%3B2-S SIAM Review is currently published by Society for Industrial and Applied Mathematics. Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at http://www.jstor.org/about/terms.html. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use. Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at http://www.jstor.org/journals/siam.html. Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission. The JSTOR Archive is a trusted digital repository providing for long-term preservation and access to leading academic journals and scholarly literature from around the world. The Archive is supported by libraries, scholarly societies, publishers, and foundations. It is an initiative of JSTOR, a not-for-profit organization with a mission to help the scholarly community take advantage of advances in technology. For more information regarding JSTOR, please contact [email protected]. http://www.jstor.org Fri Aug 17 11:22:18 2007 SIAM REVIEW 0 1989 Society for Industrial and Applied Mathematics Vol.
    [Show full text]
  • William Karush and the KKT Theorem
    Documenta Math. 255 William Karush and the KKT Theorem Richard W. Cottle 2010 Mathematics Subject Classification: 01, 90, 49 Keywords and Phrases: Biography, nonlinear programming, calculus of variations, optimality conditions 1 Prologue This chapter is mainly about William Karush and his role in the Karush-Kuhn- Tucker theorem of nonlinear programming. It tells the story of fundamental optimization results that he obtained in his master’s thesis: results that he neither published nor advertised and that were later independently rediscov- ered and published by Harold W. Kuhn and Albert W. Tucker. The principal result – which concerns necessary conditions of optimality in the problem of minimizing a function of several variables constrained by inequalities – first became known as the Kuhn–Tucker theorem. Years later, when awareness of Karush’s pioneering work spread, his name was adjoined to the name of the theorem where it remains to this day. Still, the recognition of Karush’s discov- ery of this key result left two questions unanswered: why was the thesis not published? and why did he remain silent on the priority issue? After learning of the thesis work, Harold Kuhn wrote to Karush stating his intention to set the record straight on the matter of priority, and he did so soon thereafter. In his letter to Karush, Kuhn posed these two questions, and Karush answered them in his reply. These two letters are quoted below. Although there had long been optimization problems calling for the maxi- mization or minimization of functions of several variables subject to constraints, it took the advent of linear programming to inspire the name “nonlinear pro- gramming.” This term was first used as the title of a paper [30] by Harold W.
    [Show full text]
  • NBS-INA-The Institute for Numerical Analysis
    t PUBUCATiONS fl^ United States Department of Commerce I I^^^V" I ^1 I National Institute of Standards and Tectinology NAT L INST. OF STAND 4 TECH R.I.C. A111D3 733115 NIST Special Publication 730 NBS-INA — The Institute for Numerical Analysis - UCLA 1947-1954 Magnus R, Hestenes and John Todd -QC- 100 .U57 #730 1991 C.2 i I NIST Special Publication 730 NBS-INA — The Institute for Numerical Analysis - UCLA 1947-1954 Magnus R. Hestenes John Todd Department of Mathematics Department of Mathematics University of California California Institute of Technology at Los Angeles Pasadena, CA 91109 Los Angeles, CA 90078 Sponsored in part by: The Mathematical Association of America 1529 Eighteenth Street, N.W. Washington, DC 20036 August 1991 U.S. Department of Commerce Robert A. Mosbacher, Secretary National Institute of Standards and Technology John W. Lyons, Director National Institute of Standards U.S. Government Printing Office For sale by the Superintendent and Technology Washington: 1991 of Documents Special Publication 730 U.S. Government Printing Office Natl. Inst. Stand. Technol. Washington, DC 20402 Spec. Publ. 730 182 pages (Aug. 1991) CODEN: NSPUE2 ABSTRACT This is a history of the Institute for Numerical Analysis (INA) with special emphasis on its research program during the period 1947 to 1956. The Institute for Numerical Analysis was located on the campus of the University of California, Los Angeles. It was a section of the National Applied Mathematics Laboratories, which formed the Applied Mathematics Division of the National Bureau of Standards (now the National Institute of Standards and Technology), under the U.S.
    [Show full text]
  • Pi Mu Epsilon Journal the Official Publication of the Honorary Mathematical Fraternity
    PI MU EPSILON JOURNAL THE OFFICIAL PUBLICATION OF THE HONORARY MATHEMATICAL FRATERNITY VOLUME 3 NUMBER 3 CONTENTS Plane Geometry Without Postulates - Leonard M. Blumenthal 101 An Appreciation of Giuseppe Peano - Hubert C. Kennedy . 107 On The Division of One Polynomial by Another - Michael C. Mackey . 114 Problem Department . 118 Book Reviews - . 122 Richard Kao, Franz E. Hohn, W. J. Poppelbaum, F. M. Woodworth, M. K. Fort, Jr., Gian-Carlo Rota, A. S. Householder, Murray S. Klamkin, Jane Robertson, Paul T. Bateman, W. A. Vezeau, Horace W. Norton Books Received for Review. 135 Operations Unlimited . 136 News and Notices . 146 Department Devoted to Chapter Activities . 153 , Initiates.. 160 FALL 19601 f .. Copyright 1960 by Pi Mu Epsilon Fraternity, Inc. PLANE GEOMETRY WITHOUT POSTULATESa LEONARD M. BLUMENTHAL Missouri Alpha, 1937 PI MU EPSILON JOURNAL Introduction. In his famous Elements, Euclid made a distinc- THE OFFICIAL PUBLICATIOJN tion, usually neglected today, between certain assumptions that OFTHEHONORARY MATHEMATICALFRATERNITY he called axioms, and others that he referred to as postulates. The former characterized those basic statements that expressed Francis Regan, Editor common notions (e.g., If equals are added to equals, the sums are equal) while the latter term was used for those assumptions that are peculiar to geometry (e.g., To describe a circle with any ASSOCIATE EDITORS center and radius). - The title of this paper is to be understood in the light of that Josephine Chanler Franz E. Hohn ancient discrimination. The basis for euclidean plane geometry Mary Cummings H. T. Kames presented here contains no postulates whatever (and consequent- M. S. Klarnkin ly is free of primitive or undefined terms), but it is not devoid of common notions.
    [Show full text]
  • NBS-INA-The Institute for Numerical Analysis
    t PUBUCATiONS fl^ United States Department of Commerce I I^^^V" I ^1 I National Institute of Standards and Tectinology NAT L INST. OF STAND 4 TECH R.I.C. A111D3 733115 NIST Special Publication 730 NBS-INA — The Institute for Numerical Analysis - UCLA 1947-1954 Magnus R, Hestenes and John Todd -QC- 100 .U57 #730 1991 C.2 i I NIST Special Publication 730 NBS-INA — The Institute for Numerical Analysis - UCLA 1947-1954 Magnus R. Hestenes John Todd Department of Mathematics Department of Mathematics University of California California Institute of Technology at Los Angeles Pasadena, CA 91109 Los Angeles, CA 90078 Sponsored in part by: The Mathematical Association of America 1529 Eighteenth Street, N.W. Washington, DC 20036 August 1991 U.S. Department of Commerce Robert A. Mosbacher, Secretary National Institute of Standards and Technology John W. Lyons, Director National Institute of Standards U.S. Government Printing Office For sale by the Superintendent and Technology Washington: 1991 of Documents Special Publication 730 U.S. Government Printing Office Natl. Inst. Stand. Technol. Washington, DC 20402 Spec. Publ. 730 182 pages (Aug. 1991) CODEN: NSPUE2 ABSTRACT This is a history of the Institute for Numerical Analysis (INA) with special emphasis on its research program during the period 1947 to 1956. The Institute for Numerical Analysis was located on the campus of the University of California, Los Angeles. It was a section of the National Applied Mathematics Laboratories, which formed the Applied Mathematics Division of the National Bureau of Standards (now the National Institute of Standards and Technology), under the U.S.
    [Show full text]
  • Conjugate Gradient and Lancz'os Algorithmsi
    Numerical Analysis Project June 1987 Manuscript NA-87-05 Some History of the -Conjugate Gradient and Lancz’os Algorithmsi 1948-1976 Gene H. Golub and Dianne P. O’Leary Numerical Analysis Project Computer Science Department Stanford University Stanford, California 94306 . &me H’iry of the co~ugafe Gradient and Lancso8 Algorithms 194.8-1976 Gene H. Golub Computer Science Department Stanford University Stanford, CA 94305 Dianne P. O’Leary Computer Science Department and Institute for Advanced Computer Studies University of Maryland College Park, MD 20742 ABSTRACT This manuscript gives some of the history of the conjugate gradient and Lanczos algorithms and an annotated bibliography for the period 1949-1976. This work was supported in part by the National Science Foundation under grant DCR 84-12314 (Golub) and the Air Force Of&e of Scientific Research under grant AFOSR 82-0078 (O’Leary). cor4juga~ Gradient and hncros H’ihmy 2 . 1. Introduction The conjugate gradient and Lancaos algorithms for solving linear systems of equations and eigenproblems represent very important computational innovations of the early 1950’s. These methods came into wide use only in the mid-1970’s. Shortly thereafter, vector computers and massive computer .memories made it possible to use these methods to solve problems which could not be solved in any other way. Since that time, the algorithms have been further refined and have become a basic tool for solving a wide variety of problems on a wide variety of computer architectures. The conjugate gradient algorithm has also been extended to solve nonlinear sys- tems of equations and optimization problems, and thii has had tremendous impact on the compu- tation of unconstrained and constrained optimization problems.
    [Show full text]
  • Perspectives on American Mathematics
    BULLETIN (New Series) OF THE AMERICAN MATHEMATICAL SOCIETY Volume 37, Number 4, Pages 381{405 S 0273-0979(00)00873-9 Article electronically published on June 26, 2000 PERSPECTIVES ON AMERICAN MATHEMATICS KAREN HUNGER PARSHALL Abstract. A research-level community of mathematicians developed in the United States in the closing quarter of the nineteenth century. Since that time, American mathematicians have regularly paused to assess the state of their community and to reflect on its mathematical output. This paper analyzes a series of such reflections|beginning with Simon Newcomb's thoughts on the state of the exact sciences in America in 1874 and culminating with the 1988 commentaries on the \problems of mathematics" discussed at Princeton's bicentennial celebrations in 1946|against a backdrop of broader historical trends. \The prospect of mathematics [in the United States] is about as discouraging as the retrospect" [46, p. 288]. So wrote mathematical astronomer Simon Newcomb in an 1874 essay on the state of the exact sciences in America. He was no more optimistic two years later in his assessment of the abstract sciences at the time of the United States' centennial. \When we inquire into the wealth and power of our scientific organizations, and the extent of their publications|when, in fact, we consider merely the gross quantity of original published research," he wrote, \we see our science in the aspect best fitted to make us contemplate the past with humility and the future with despair" [45, p. 116]. In some sense, Newcomb had every reason to be pessimistic about the devel- opment of mathematics and of the mathematical sciences in the United States in the early 1870s.
    [Show full text]
  • Program of the Sessions San Diego, California, January 10–13, 2018
    Program of the Sessions San Diego, California, January 10–13, 2018 Monday, January 8 AMS Short Course Reception AMS Short Course on Discrete Differential 5:00 PM –6:00PM Room 5B, Upper Level, Geometry, Part I San Diego Convention Center 8:00 AM –5:00PM Room 5A, Upper Level, San Diego Convention Center 8:00AM Introduction & Overview Tuesday, January 9 8:30AM Discrete Laplace operators - theory. (1) Max Wardetzky, University of Goettingen AMS Short Course on Discrete Differential 9:30AM Break Geometry, Part II 9:50AM Discrete Laplace operators - applications. (2) Max Wardetzky, University of Goettingen 8:00 AM –5:00PM Room 5A, Upper Level, San Diego Convention Center 10:50AM Break 11:10AM Discrete parametric surfaces - theory. 8:00AM Discrete mappings - theory. (3) Johannes Wallner,TUGraz (5) Yaron Lipman, Weizmann Institute of 12:10PM Lunch Science 1:30PM Discrete parametric surfaces - 9:00AM Break (4) applications. 9:20AM Discrete mappings - applications. Johannes Wallner,TUGraz (6) Yaron Lipman, Weizmann Institute of 2:30PM Free Time Science 3:30PM Demo Session 10:20AM Break 10:40AM Discrete conformal geometry - theory. NSF-EHR Grant Proposal Writing Workshop (7) Keenan Crane,CarnegieMellon University 3:00 PM –6:00PM Coronado Room, 4th Floor, 11:40AM Lunch South Tower, Marriott Marquis San Diego Marina 1:00PM Discrete conformal geometry - Organizers: Ron Buckmire,National (8) applications. Science Foundation Keenan Crane,CarnegieMellon Lee Zia, National Science University Foundation 2:00PM Break The time limit for each AMS contributed paper in the sessions meeting will be found in Volume 39, Issue 1 of Abstracts is ten minutes. The time limit for each MAA contributed of papers presented to the American Mathematical Society, paper varies.
    [Show full text]
  • Engineering Optimization: an Introduction with Metaheuristic Applications
    Engineering Optimization An Introduction with Metaheuristic Applications Xin-SheYang University of Cambridge Department of Engineering Cambridge, United Kingdom WILEY A JOHN WILEY & SONS, INC., PUBLICATION Thispageintentionallyleftblank Engineering Optimization Thispageintentionallyleftblank Engineering Optimization An Introduction with Metaheuristic Applications Xin-SheYang University of Cambridge Department of Engineering Cambridge, United Kingdom WILEY A JOHN WILEY & SONS, INC., PUBLICATION Copyright © 2010 by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as per­ mitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be ad­ dressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or com­ pleteness of the contents of this book and specifically disclaim any implied warranties of merchantabili­ ty or fitness for a particular purpose.
    [Show full text]
  • Some History of Conjugate Gradients and Other Krylov Subspace Methods October 2009 SIAM Applied Linear Algebra Meeting 2009 Dianne P
    Some History of Conjugate Gradients and Other Krylov Subspace Methods October 2009 SIAM Applied Linear Algebra Meeting 2009 Dianne P. O'Leary c 2009 1 Some History of Conjugate Gradients Dianne P. O'Leary Computer Science Dept. and Institute for Advanced Computer Studies University of Maryland [email protected] http://www.cs.umd.edu/users/oleary 2 A Tale of 2 Cities Los Angeles, California Z¨urich,Switzerland 3 ... and a Tale of 2 Men 4 ... or maybe 3 Men 5 ... or maybe 5 Men 6 The Conjugate Gradient Algorithm 7 Notation • We solve the linear system Ax∗ = b where A 2 Rn×n is symmetric positive definite and b 2 Rn. • We assume, without loss of generality, that our initial guess for the solution is x(0) = 0 : • We denote the Krylov subspace of dimension m by m−1 Km(A; b) = spanfb; Ab; : : : ; A bg : (k) • Then the conjugate gradient algorithm chooses x 2 Kk(A; b) to minimize (x − x∗)T A(x − x∗). { work per iteration: 1 matrix-vector multiply and a few dot products and combinations of vectors. { storage: 3 vectors, plus original data. 8 Context for the Conjugate Gradient Algorithm Hestenes and Stiefel (1952) presented the conjugate gradient algorithm in the Journal of Research of the NBS. Context: Two existing classes of algorithms • Direct methods, like Gauss elimination, modified a tableau of matrix entries in a systematic way in order to compute the solution. These methods were finite but required a rather large amount of computational effort, with work growing as the cube of the number of unknowns.
    [Show full text]
  • Engineering Optimization
    Engineering Optimization An Introduction with Metaheuristic Applications Xin-SheYang University of Cambridge Department of Engineering Cambridge, United Kingdom WILEY A JOHN WILEY & SONS, INC., PUBLICATION Thispageintentionallyleftblank Engineering Optimization Thispageintentionallyleftblank Engineering Optimization An Introduction with Metaheuristic Applications Xin-SheYang University of Cambridge Department of Engineering Cambridge, United Kingdom WILEY A JOHN WILEY & SONS, INC., PUBLICATION Copyright © 2010 by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as per­ mitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be ad­ dressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or com­ pleteness of the contents of this book and specifically disclaim any implied warranties of merchantabili­ ty or fitness for a particular purpose.
    [Show full text]