Functional Analysis and Optimization Kazufumi Ito∗ November 29, 2016 Abstract In this monograph we develop the function space method for optimization problems and operator equations in Banach spaces. Optimization is the one of key components for mathematical modeling of real world problems and the solution method provides an accurate and essential description and validation of the mathematical model. Op- timization problems are encountered frequently in engineering and sciences and have widespread practical applications.In general we optimize an appropriately chosen cost functional subject to constraints. For example, the constraints are in the form of equal- ity constraints and inequality constraints. The problem is casted in a function space and it is important to formulate the problem in a proper function space framework in order to develop the accurate theory and the effective algorithm. Many of the constraints for the optimization are governed by partial differential and functional equations. In order to discuss the existence of optimizers it is essential to develop a comprehensive treatment of the constraints equations. In general the necessary optimality condition is in the form of operator equations. Non-smooth optimization becomes a very basic modeling toll and enlarges and en- hances the applications of the optimization method in general. For example, in the classical variational formulation we analyze the non Newtonian energy functional and nonsmooth friction penalty. For the imaging/signal analysis the sparsity optimization is used by means of L1 and TV regularization. In order to develop an efficient solu- tion method for large scale optimizations it is essential to develop a set of necessary conditions in the equation form rather than the variational inequality. The Lagrange multiplier theory for the constrained minimizations and nonsmooth optimization prob- lems is developed and analyzed The theory derives the complementarity condition for the multiplier and the solution. Consequently, one can derive the necessary optimality system for the solution and the multiplier for a general class of nonsmooth and noncovex optimizations. In the light of the theory we derive and analyze numerical algorithms based the primal-dual active set method and the semi-smooth Newton method. The monograph also covers the basic materials for real analysis, functional anal- ysis, Banach space theory, convex analysis, operator theory and PDE theory, which makes the book self-contained, comprehensive and complete. The analysis component is naturally connected to the optimization theory. The necessary optimality condition is in general written as nonlinear operator equations for the primal variable and La- grange multiplier. The Lagrange multiplier theory of a general class of nonsmooth and ∗Department of Mathematics, North Carolina State University, Raleigh, North Carolina, USA 1 non-convex optimization are based on the functional analysis tool. For example, the necessary optimality for the constrained optimization is in general nonsmooth and a psuedo-monontone type operator equation. To develop the mathematical treatment of the PDE constrained minimization we also present the PDE theory and linear and nonlinear C0-semigroup theory on Banach spaces and we have a full discussion of well-posedness of the control dynamics and optimization problems. That is, the existence and uniqueness of solutions to a general class of controlled linear and nonlinear PDEs and Cauchy problems for nonlinear evo- lution equations in Banach space are discussed. Especially the linear operator theory and the (psuedo-) monotone operator and mapping theory and the fixed point the- ory.Throughout the monograph demonstrates the theory and algorithm using concrete examples and describes how to apply it for motivated applications 1 Introduction In this monograph we develop the function space approach for the optimization prob- lems, e.g., nonlinear programming in Banach spaces, convex and non-convex nonsmooth variational problems, control and inverse problems, image/signal analysis, material design, classification, and resource allocation. We also develop the basic functional analysis tool for for the nonlinear equations and Cauchy problems in Banach spaces in order to establish the well-posedness of the constraint optimization. The function space optimization method allows us to study the well-posednees for a general class of optimization problems systematically using the functional analysis tool and it also enables us to develop and analyze the numerical optimization algorithm based on the calculus of variations and the Lagrange multiplier theory for the constrained opti- mization. The function space analysis provides the regularizing procedure to remeady the ill-posedness of the constraints and the necessary optimality system and thus one can develop a stable and effective numerical algorithm. The convergence analysis for the function space optimization guides us to analyze and improve the stability and effectiveness of algorithms for the associated large scale optimizations. A general class of constrained optimization problems in function spaces is consid- ered and we develop the Lagrange multiplier theory and effective solution algorithms. Applications are presented and analyzed for various examples including control and design optimization, inverse problems, image analysis and variational problems. Non- convex and non-smooth optimization becomes increasing important for the advanced and effective use of optimization methods and thus they are essential topics for the monograph. We introduce the basic function space and functional analysis tools needed for our analysis. The monograph provides an introduction to the functional analysis, real analysis and convex analysis and their basic concepts with illustrated examples. Thus, one can use the book as a basic course material for the functional analysis and nonlinear operator theory. The nonlinear operator theory and their applications to PDE's problems are presented in details, including classical variational optimization problems in Newtonian and non-Newtonian mechanics and fluid. An emerging appli- cation of optimizations include the imaging and signal analysis and the classification and machine learning. It is often formulated as the PDE's constrained optimization. There are numerous applications of the optimization theory and variational method in all sciences and to real world applications. For example, the followings are the 2 motivated examples. Example 1 (Quadratic programming) 1 t t n min 2 x Ax − a x over x 2 R subject to Ex = b (equality constraint) Gx ≤ c (inequality constraint): where A 2 Rn×n is a symmetric positive matrix, E 2 Rm×n;G 2 Rp×n, and a 2 Rn, b 2 Rm, c 2 Rp are given vectors. The quadratic programming is formulated on a Hilbert space X for the variational problem, in which (Ax; x)X defines the quadratic form on X. Example 2 (Linear programming) The linear programing minimizes (c; x)X subject to the linear constraint Ax = b and x ≥ 0. Its dual problem is to maximize (b; x) subject to A∗x ≤ c. It has many important classes of applications. For example, the optimal transport problem is to find the joint distribution function µ(x; y) ≥ 0 on X × Y that transports the probability density p0(x) to p1(y), i.e. Z Z µ(x; y) dx = p0(x); µ(x; y) dy = p1(y); Y Y with minimum energy Z c(x; y)µ(x; y) dxdy = (c; µ): X×Y Example 3 (Integer programming) Let F is a continuous function Rn. The integer programming is to minimize F (x) over the integer variables x, i.e., min F (x) subject to x 2 f0; 1; ··· Ng: The binary optimizations to minimize F (x) over the binary variables x. The network optimization problem is the one of important applications. Example4 (Obstacle problem) Consider the variational problem; Z 1 1 d min J(u) = ( j u(x)j2 − f(x)u(x)) dx (1.1) 0 2 dx subject to u(x) ≤ (x) 1 over all displacement function u(x) 2 H0 (0; 1), where the function represents the upper bound (obstacle) of the deformation. The functional J represents the Newtonian deformation energy and f(x) is the applied force. Example 5 (Function Interpolation) Consider the interpolation problem; Z 1 2 1 d 2 min ( j 2 u(x)j dx over all functions u(x); (1.2) 0 2 dx subject to the interpolation conditions d u(x ) = b ; u(x ) = c ; 1 ≤ i ≤ m: i i dx i i 3 Example 6 (Image/Signal Analysis) Consider the deconvolution problem of finding the image u that satisfies (Ku)(x) = f(x) for the convolution K Z (Ku)(x) = k(x; y)u(y) dy: Ω over a domain Ω. It is in general ill-posed and we use the (Tikhonov) regularization formulation to have a stable deconvolution solution u, i.e., for Ω = [0; 1] Z 1 α d min (jKu − fj2 + j u(x)j2 + β u(x)) dx over all possible image u ≥ 0; (1.3) 0 2 dx where α; β ≥ 0 are the regularization parameters. The first regularization term is the quadratic varation of image u. Since if u ≥ 0 then juj = u, the regularization term for the L1 sparsity. The two regularization parameters are chosen so that the variation of image u is regularized. Example 7 (L0 optimization) Problem of optimal resource allocation and material dis- tribution and optimal time scheduling can be formulated as Z F (u) + h(u(!)) d! (1.4) Ω α 2 0 with h(u) = 2 juj + β juj where j0j0 = 0; juj0 = 1; otherwise: Since Z ju(!)j0 d! = vol(fu(!) 6= 0g); Ω the solution to (1.4) provides the optimal distribution of u for minimizing the perfor- mance F (u) with appropriate choice of (regularization) parameters α; β > 0. Example 8 (Bayesian statistics) Consider the statical optimization; min −log (p(yjx) + βp(x)) over all admissible parameters x; (1.5) where p(yjx) is the conditional probability density of observation y given parameter x and the p(x) is the prior probability density for parameter x.