Noisy Information and Computational Complexity
Total Page:16
File Type:pdf, Size:1020Kb
Noisy Information and Computational Complexity L. Plaskota Institute of Applied Mathematics and Mechanics University of Warsaw Contents 1 Overview 5 2 Worst case setting 9 2.1 Introduction . 9 2.2 Information, algorithm, approximation . 12 2.3 Radius and diameter of information . 17 2.4 Affine algorithms for linear functionals . 24 2.4.1 Existence of optimal affine algorithms . 25 2.4.2 The case of Hilbert noise . 28 2.5 Optimality of spline algorithms . 37 2.5.1 Splines and smoothing splines . 38 2.5.2 α{smoothing splines . 41 2.6 Special splines . 50 2.6.1 The Hilbert case with optimal α . 51 2.6.2 Least squares and regularization . 57 2.6.3 Polynomial splines . 61 2.6.4 Splines in r.k.h.s. 65 2.7 Varying information . 71 2.7.1 Nonadaptive and adaptive information . 71 2.7.2 When does adaption not help? . 74 2.8 Optimal information . 78 2.8.1 Linear problems in Hilbert spaces . 79 2.8.2 Approximation and integration of Lipschitz functions 89 2.9 Complexity . 96 2.9.1 Computations over the space G . 97 2.9.2 Cost and complexity, general bounds . 99 2.10 Complexity of special problems . 107 2.10.1 Linear problems in Hilbert spaces . 107 1 2 CONTENTS 2.10.2 Approximation and integration of Lipschitz functions 114 2.10.3 Multivariate approximation in a Banach space . 117 3 Average case setting 129 3.1 Introduction . 129 3.2 Information and its radius . 131 3.3 Gaussian measures on Banach spaces . 140 3.3.1 Basic properties . 140 3.3.2 Gaussian measures as abstract Wiener spaces . 142 3.4 Linear problems with Gaussian measures . 147 3.4.1 Induced and conditional distributions . 148 3.4.2 Optimal algorithms . 151 3.5 The case of linear functionals . 157 3.6 Optimal algorithms as smoothing splines . 164 3.6.1 A general case . 164 3.6.2 Special cases . 165 3.6.3 A correspondence theorem . 168 3.7 Varying information . 170 3.7.1 Nonadaptive and adaptive information . 170 3.7.2 Adaption versus nonadaption, I . 173 3.8 Optimal information . 177 3.8.1 Linear problems with Gaussian measures . 178 3.8.2 Approximation and integration on the Wiener space . 188 3.9 Complexity . 206 3.9.1 Adaption versus nonadaption, II . 207 3.9.2 General bounds . 211 3.10 Complexity of special problems . 215 3.10.1 Linear problems with Gaussian measures . 215 3.10.2 Approximation and integration on the Wiener space . 221 4 First mixed setting 227 4.1 Introduction . 227 4.2 Affine algorithms for linear functionals . 228 4.2.1 The one{dimensional problem . 229 4.2.2 Almost optimality of affine algorithms . 233 4.2.3 A correspondence theorem . 242 4.3 Approximation of operators . 246 4.3.1 Ellipsoidal problems in Rn . 246 4.3.2 The Hilbert case . 250 CONTENTS 3 5 Second mixed setting 259 5.1 Introduction . 259 5.2 Linear algorithms for linear functionals . 260 5.2.1 The one{dimensional problem . 260 5.2.2 Almost optimality of linear algorithms . 264 5.2.3 A correspondence theorem . 272 5.3 Approximation of operators . 274 6 Asymptotic setting 279 6.1 Introduction . 279 6.2 Asymptotic and worst case settings . 280 6.2.1 Information, algorithm and error . 281 6.2.2 Optimal algorithms . 282 6.2.3 Optimal information . 288 6.3 Asymptotic and average case settings . 292 6.3.1 Optimal algorithms . 293 6.3.2 Convergence rate . 296 6.3.3 Optimal information . 301 4 CONTENTS Chapter 1 Overview In the process of doing scientific computations we always rely on some infor- mation. A typical situation in practice is that this information is contami- nated by errors. We say that it is noisy. Sources of noise include: previous computations, • inexact measurements, • transmission errors, • arithmetic limitations, • adversary's lies. • Problems with noisy information have always attracted a considerable attention of researchers in many different scientific fields: statisticians, engi- neers, control theorists, economists, applied mathematicians. There is also a vast literature, especially in statistics, where noisy information is analyzed from different perspectives. In this monograph, noisy information is studied in the context of the computational complexity of solving mathematically posed problems. The computational complexity focuses on the intrinsic difficulty of prob- lems as measured by the minimal amount of time, memory, or elementary operations necessary to solve them. Information{based complexity (IBC) is a branch of computational complexity that deals with problems for which the available information is: partial, • noisy, • priced. • 5 6 CHAPTER 1. OVERVIEW Information being partial means that the problem is not uniquely de- termined by the given information. Information is noisy since it may be contaminated by some errors. Finally, information is priced since we must pay for getting it. These assumptions distinguish IBC from combinatorial complexity, where information is complete, exact, and free. Since information is partial and noisy, only approximate solutions are possible. One of the main goals of IBC is finding the complexity of the prob- lem, i.e., the intrinsic cost of computing an approximation with given accu- racy. Approximations are obtained by algorithms that use some information. These solving the problem with minimal cost are of special importance and called optimal. Partial, noisy and priced information is typical of many problems arising in different scientific fields. These include, for instance, signal processing, control theory, computer vision, and numerical analysis. As a rule, a digital computer is used to perform scientific computations. A computer can only make use of a finite set of numbers. Usually, these numbers cannot be exactly entered into the computer memory. Hence, problems described by infinitely many parameters can be \solved" using only partial and noisy information. The theory of optimal algorithms for solving problems with partial in- formation has a long history. It can be traced back to the late forties when Kiefer, Sard and Nikolskij wrote pioneering papers. A systematic and uni- form approach to such kind of problems was first presented by J.F. Traub and H. Wo´zniakowski in the monograph A General Theory of Optimal Al- gorithms, Academic Press, 1980. This was an important stage in the devel- opment of the theory of IBC. The monograph was followed then by Information, Uncertainty, Com- plexity, Addison-Wesley, 1983, and Information-Based Complexity, Academic Press, 1988, both authored by J.F. Traub, G.W. Wasilkowski, and H. Wo´znia- kowski. Computational complexity of approximately solved problems is also studied in the books: Deterministic and Stochastic Error Bounds in Numer- ical Analysis by E. Novak, Springer Verlag, 1988, and The Computational Complexity of Differential and Integral Equations by A.G. Werschulz, Oxford University Press, 1991. Relatively few IBC papers study noisy information. One reason is the technical difficulty of the analysis of noisy information. A second reason is that even if we are primarily interested in noisy information, the results on exact information establish a benchmark. All negative results for exact information are also applicable for the noisy case. On the other hand, it is not clear whether positive results for exact information have a counterpart 7 for noisy information. In the mathematical literature, the word \noise" is used mainly by statis- ticians and means a random error that occurs for experimental observations. We also want to study deterministic error. Therefore by noise, we mean random or deterministic error. Moreover, in our model, the source of the information is not important. We may say that \information is observed" or that it is \computed". We also stress that the case of exact information is not excluded, neither in the model nor in most results. Exact information is obtained as a special case by setting the noise level to zero. This permits us to study the depen- dence of the results on the noise level, and to compare the noisy and exact information cases. In general, optimal algorithms and problem complexity depend on the setting. The setting is specified by the way the error and cost of an algorithm are defined. If the error and cost are defined by their worst performance, we have the worst case setting. The average case setting is obtained when the average performance of algorithms is considered. In this monograph, we study the worst and average case settings as well as mixed settings and asymptotic setting. Other settings such as probabilistic and randomized settings will be the topic of future research. Despite the differences, the settings have certain features in common. For instance, algorithms that are based on smoothing splines are optimal, independent of the setting. This is a very desirable property, since it shows that such algorithms are universal and robust. Most of the research presented in this monograph has been done over the last 5{6 years by different people, including the author. Some of the results have not been previously reported. The references to the original results are given in Notes and Remarks at the end of each section. Clearly, the author does not pretend to cover the whole subject of noisy information in one monograph. Only these topics are presented that are typical of IBC, or are needed for the complexity analysis. Many problems are still open. Some of these are indicated in the text. The monograph consists of six chapters. We start with the worst case setting in Chapter 2. Chapter 3 is devoted to the average case setting. Each of these two settings is studied following the same scheme. We first look for the best algorithms that use fixed information. Then we allow the informa- tion to vary and seek optimal information.