1.1 1.2 1. Introduction then a global minimum exists. If W = (a, b) then minima either exist or not, depending on the choice of a, b. In general, a continuous function is not Optimization problem guaranteed to have extrema if the feasible set is not compact, e.g. if it is open. Let W Rn be a nonempty set and f: W R a function. We consider the Existence of minima of a continuous function ⊂ problem of finding minima of f in W, taking in particular → Theorem 1 If the set W Rn is compact and f: W R is a continuous n ∈ ¯ W = R (unconstrained optimization), function, then f reaches its infimum and supremum in W, i.e., there exist x0, y0 W such that → n ∈ ¯ W = { x R : g (x)= 0,...,g (x)= 0 }, where g ,...,g are functions ∈ 1 m 1 m Rn Rn f(x0) 6 f(x) 6 f(y0) for all x W. (equality constraints), ∈ n ¯ W = { x R : g (x) 6 0,...,g (x) 6 0 }, where g ,...,g are functions → 1 m 1 m Rn Rn∈ Definition 3 A function f: W R is called coercive if f(x) for x . (inequality constraints). k k Equivalently, → → ∞ → ∞ → x >s f(x) > r. The set W is called a feasible set/region. ∀r>0∃s>0∀x∈W k k ⇒ If W is a bounded set, then any function f: W R is coercive. Definition 1 A point x W is called a global minimum of f in W if 0 ∈ n → f(x) > f(x0) for all x W. Theorem 2 If W R is a closed set and f: W R is continuous and ∈ ⊂ coercive, then there exists a minimum x0 of f in W. → Definition 2 A point x W is called a local minimum of f in W if there 0 ∈ exists ε>0 such that Proof. For a point y W we define the set Uy = { x W : f(x) 6 f(y) }. The set ∈ ∈ Uy is nonempty and closed, as the function f is continuous and the inequality in f(x) > f(x0) for all x W B(x0,ε), the definition of U is nonsharp and W is closed. This set is also bounded: for ∈ ∩ y r = f(y), from the coercivity of f there exists s>0 such that if x >s, then where B(x ,ε) is the ball whose centre is x and the radius is ε. k k 0 0 f(x) > r = f(y); hence, x / U and U B(0,s). It follows that U is a closed ∈ y y ⊂ y and bounded set, i.e., it is compact. Therefore there exists a global minimum x0 of f in U . Due to f(x) >f(y) > f(x ) for x / U , x is also a global minimum of f Any global minimum is a local minimum. A minimum is called strict if in the y 0 ∈ y 0 in W. ✷ definitions above there is f(x) >f(x ) for x = x . In a similar way we define global 0 6 0 and local maxima. A point x0 is a (global or local) extremum if it is a minimum or a maximum. Theorem 3 Let W Rn be nonempty and let f: W R be a continuous ⊂ function. If there exists y W such that for any sequence (x ) W such ∈ n n ⊂ Minima need not exist, if no point x0 fulfills the definitions. A global minimum that → does not exist if infx∈W f(x)=− or infx∈W f(x)= c and f(x) > c for all x W. ∈ x cl W \ W or x n k nk Example. Let f(x)= x cos x. If W∞= R then inf f(x)=− , and there is no there is lim inf f(x ) >f(y), then there exists a minimum x of the x∈W → n→∞ n → ∞ 0 global minimum and an infinite set of local minima. If W = [a, b], where a, b R, function f. ∈ ∞ 1.3 1.4 Proof. The set Uy is defined as before. To show that it is closed, we take any Taylor’s formulae sequence (x ) U which converges to x. It suffices to show that x U . From n n ⊂ y ∈ y x U we have f(x ) 6 f(y) and if x / W, then we have an inconsistency with n ∈ y n ∈ the assumption. Hence, x W. As the function f is continuous in W, there is Theorem 8 (Rolle’s theorem) If a function f: [a, b] R is continuous ∈ f(x) 6 f(y), hence x Uy. The set Uy is also bounded, which follows from the in [a, b], differentiable in (a, b) and f(a)= f(b), then there exists a point ∈ ′ assumed implication x lim inf f(x ) >f(y). The proof is x0 (a, b) such that f (x0)= 0. → k nk n→∞ n ∈ completed just like the proof of the previous theorem. ✷ → ∞ ⇒ Local minima of functions of one variable Proof. If f is constant, then the claim is obvious. Otherwise there exists an extremum x0 of f in [a, b] other than a and b: there is f(x )= f(x) >f(a) f(x )= f(x) <f(a) x Let W R be an open set. 0 supx∈[a,b] or 0 infx∈[a,b] . Let 0 be a maximum. ⊂ Then f(x) 6 f(x ) for all x [a, b] and 0 ∈ f(x)− f(x0) f(x)− f(x0) Theorem 4 (necessary condition of the 1st order) If x0 W is a local > 0 if x<x0, 6 0 if x>x0. ′ ′ ∈ x − x0 x − x0 minimum or maximum of f and f (x0) exists, then f (x0)= 0. Hence, Proof. Let x0 be a local minimum. For sufficiently small h>0 there is ′ f(x)− f(x0) f(x)− f(x0) f (x0)= lim = lim , f(x0 − h) > f(x0) 6 f(x0 + h) and then xրx0 x − x0 xցx0 x − x0 >0 60 f(x0 − h)− f(x0) f(x0 − h)− f(x0) ′ 6 0 lim 6 0 f (x0) 6 0, −h h→0 −h ′ therefore, f (x0)= |0. If {zx0 is a} minimum,| the{z proof} is similar. ✷ f(x0 + h)− f(x0) f(x0 + h)− f(x0) ′ > 0 ⇒ lim > 0 ⇒ f (x0) > 0, +h h→0 +h hence, f′(x )= 0. ✷ 0 ⇒ ⇒ Theorem 9 (mean value theorem) If a function f: [a, b] R is continuous in [a, b] and differentiable in (a, b), then there exists a point x (a, b) such 0 ∈ that Theorem 5 (necessary condition of the 2nd order) If f: W R is of → 2 ′′ class C (W) and x0 is a local minimum, then f (x0) > 0. ′ f(b)− f(a)= f (x0)(b − a). → If the set W is not open, then we cannot use the above theorems for x0 ∂W. But ∈ def the theorem below applies also in this case. Proof. Let g(x) = f(b)− f(a) x − (b − a)f(x). The function g is continuous in [a, b] and differentiable in (a, b), moreover, Theorem 6 (sufficient condition of the 2nd order) If f: W R is of g(a)= f(b)a − f(a)b = g(b). 2 ′ ′′ class C (W) and f (x0)= 0, f (x0) >0 at a point x0 W, then f has a strict ∈ ′ x By Rolle’s theorem, there exists x0 (a, b) such that g (x0)= 0. Hence, local minimum at 0. → ∈ ′ ′ 0 = g (x0)= f(b)− f(a) − (b − a)f (x0). Theorem 7 If W R is open, f Ck(W) and ⊂ ∈ ✷ f′(x )= f′′(x )= = f(k−1)(x )= 0, f(k)(x ) = 0 for x W, then if k is odd, The proof is completed by rearranging this formula. 0 0 · · · 0 0 6 0 ∈ there is no extremum of f at x0, and if k is even, then there is a local (k) (k) minimum if f (x0) >0 and a local maximum if f (x0) <0. 1.5 1.6 Theorem 10 (Taylor’s formula with the remainder in Peano form) Let Theorem 11 (Taylor’s formula with the remainder in Lagrange form) Let f: [a, b] R be a function differentiable in [a, b] and twice differentiable at f: [a, b] R be a function of class Ck−1[a, b] and k times differentiable some point x (a, b). Then for all x [a, b] there is in (a, b). For x (a, b) and x [a, b] there is 0 ∈ ∈ 0 ∈ ∈ → → ′′ k−1 ′ f (x0) 2 2 (i) (k) ( )= ( )+ ( )( − )+ ( − ) + ( − ) f (x0) f (x) f x f x0 f x0 x x0 x x0 o x x0 . f(x)= f(x )+ (x − x )i + (x − x )k, 2 0 i! 0 k! 0 i=1 X where x is a point between x and x. Proof. Without loss of generality we assume x0 = 0. Let 0 ′′ def ′ f (0) 2 R(x) = f(x)− f(0)− f (0)x − x . def (i) 2 k−1 f (x0) i Proof. The function h(x) = f(x0)+ i=1 i! (x − x0) is a polynomial of degree 2 ′ def k f(x)−h(x) We need to show that R(x)= o(x ). From the continuity of f we obtain less than k. For x = x0 let gx(y) = f(y)− h(y)− zx(y − x0) , where zx = k . 6 P (x−x0) x ′ (k−1) It is easy to verify that gx(x0)= g (x0)= = gx (x0)= gx(x)= 0.