Submitted by Kemal Raik, MA MSc. Submitted at Industrial Mathematics Institute Supervisor and First Examiner Priv. Doz. DI Dr Stefan Kindermann Second Examiner Linear and Nonlinear Univ.-Prof. Dr Bernd Hofmann Heuristic Regularisation July 2020 for Ill-Posed Problems Doctoral Thesis to obtain the academic degree of Doktor der technischen Wissenschaften in the Doctoral Program Technische Wissenschaften JOHANNES KEPLER UNIVERSITY LINZ Altenbergerstraße 69 4040 Linz, Osterreich¨ www.jku.at DVR 0093696 Abstract In this thesis, we cover the so-called heuristic (aka error-free or data-driven) parameter choice rules for the regularisation of ill-posed problems (which just so happen to be prominent in the treatment of inverse problems). We consider the linear theory associated with both continuous regularisation methods, such as that of Tikhonov, and also iterative procedures, such as Landweber's method. We provide background material associated with each of the aforementioned regularisation methods as well as the standard results found in the literature. In particular, the convergence theory for heuristic rules is typically based on a noise-restricted analysis. We also introduce some more recent developments in the linear theory for certain instances: in case of operator perturbations or weakly bounded noise for linear Tikhonov regularisation. In both the aforementioned cases, novel parameter choice rules were derived; for the case of weakly bounded noise, through necessity and in the case of operator perturbations, an entirely new class of parameter choice rules are discussed (so-called semi-heuristic rules which could be said to be the \middle ground" between heuristic rules and a-posteriori rules). We then delve further into the abyss of the relatively unknown; namely the nonlinear theory (by which we mean that the regularisation is nonlinear) for which the development and analysis of heuristic rules are still in their infancy. Most notably in this thesis, we present a recent study of the conver- gence theory for heuristic Tikhonov regularisation with convex penalty terms which attempts to generalise, to some extent, the restricted noise analysis of the linear theory. As the error in this setting is measured in terms of the Bregman distance, this naturally lends itself to the introduction of some novel parameter choice rules. Finally, we illustrate and supplement most of the preceding by including a numerics section which displays the effectiveness of heuristic parameter choice rules and conclude with a discussion of the results as well as a specu- lation of the potential future scope of research in this exciting area of applied mathematics. 1 Zusammenfassung In dieser Dissertation behandeln wir sogenannte heuristische Parameterwahl- regeln (auch noisefreie oder datengesteuerte Parameterwahlregeln genannt) fur¨ die Regularisierung schlecht gestellter Probleme (welche bei der Behand- lung von inversen Problemen eine herausragende Rolle spielen). Wir behan- deln zun¨achst lineare inverse Probleme sowohl in Kombination mit kontinu- ierlichen Regularisierungsmethoden, wie zum Beispiel der Tichonov Regulari- sierung, als auch mit iterativen, wie etwa dem Landweber Verfahren. Wir lie- fern Hintergrundmaterial zu jedem dieser Regularisierungsmethoden als auch die dazugeh¨origen Standardresultate aus der Literatur, wie etwa die Kon- vergenztheorie fur¨ heuristische Parameterwahlregeln, die typischerweise auf einer Analysis mit Einschr¨ankungen an den Datenfehler basieren. Außerdem stellen wir einige neuere Entwicklungen in der linearen Theorie fur¨ bestimmte Spezialf¨alle vor: im Fall von Operatorst¨orungen oder schwach beschr¨ankten Datenfehlern fur¨ lineare Tichonovregularisierung. In beiden F¨allen werden neuartige Parameterwahlen vorgestellt; im Fall von schwach beschr¨ankten Datenfehlern aus Notwendigkeit, und im Fall von Operatorst¨orungen wird eine v¨ollig neue Klasse von Parameterwahlregeln diskutiert (sogenannte semi- heuristische Regeln, die in gewisser Weise ein \Mittelweg" zwischen heuristi- schen und a-posteriori-Regeln sind). Anschließend tauchen wir weiter in den Abgrund des relativ Unbekannten ein, n¨amlich in die nichtlineare Theorie (d.h., wenn die Regularisierungmethode nichtlinear ist), fur¨ die Entwicklung und Analyse heuristischer Regeln noch im Kindheitsstadium sind. Bemer- kenswert in dieser Arbeit ist, dass wir eine aktuelle Konvergenztheorie fur¨ heuristische Tikhonov-Regularisierung mit konvexem Strafterm entwickeln, die versucht, die Konvergenzanalyse mit Datenfehlerbeschr¨ankungen der li- nearen Theorie bis zu einem gewissen Grad zu verallgemeinern. Da der Fehler bei diesen Methoden ublicherweise¨ in der Bregman-Distanz gemessen wird, bieten sich dementsprechend einige neuartige Regeln fur¨ die Parameterwahl an. Schließlich veranschaulichen und erg¨anzen wir die meisten der vorher- gehenden Resultate in einem Abschnitt mit numerischen Experimenten, der die Wirksamkeit heuristischer Parameterwahlen illustriert, und wir schlie- ßen mit einer Diskussion der Ergebnisse sowie Spekulationen uber¨ m¨ogliche zukunftige¨ Forschungsinhalte in diesem spannenden Anwendungsbereich der Mathematik. 2 Acknowledgements First and foremost, I would like to acknowledge and thank my supervisor, Dr Stefan Kindermann, who provided significant guidance over the course of my doctoral studies. The research topic, on which this thesis is based, was through his proposal, which was granted funding from the Austrian Science Fund (FWF) to whom I also extend my thanks. Moreover, much of the contents of this thesis are based on research which was jointly conducted by myself and my supervisor. I would also like to thank Professor Bernd Hofmann for agreeing to be the second examiner for this thesis and also for being the original organiser of the Chemnitz Symposium on Inverse Problems which I have twice had the pleasure of participating in. I also owe a great deal of thanks to my family in London for their continued support whilst I have been in Linz, particularly my mother who has visited me here on a great deal of occasions. Given the natural beauty of Austria, she did not need much convincing, however. Finally, I would like to thank my friends and colleagues; most notably, and in no particular order, Dr Simon Hubmer, Fabian Hinterer, Onkar Sandip Jadhav, Alexander Ploier and Dr G¨unter Auzinger for their friendship and discussions, both academic and otherwise. Kemal Raik Linz, July 2020 3 Contents 1 Introduction 7 1.1 Examples . .7 1.2 Preliminaries . 10 1.3 Regularisation Methods . 13 1.3.1 Continuous Methods . 14 1.3.2 Iterative Methods . 20 1.3.3 Parameter Choice Rules . 21 1.4 Heuristic Parameter Choice Rules . 23 I Theory 30 2 Linear Tikhonov Regularisation 31 2.1 Classical Theory . 31 2.1.1 Heuristic Parameter Choice Rules . 36 2.2 Weakly Bounded Noise . 52 2.2.1 Modified Parameter Choice Rules . 54 2.2.2 Predictive Mean-Square Error . 61 2.2.3 Generalised Cross-Validation . 65 2.3 Operator Perturbations . 69 2.3.1 Semi-Heuristic Parameter Choice Rules . 71 3 Convex Tikhonov Regularisation 78 3.1 Classical Theory . 78 3.2 Parameter Choice Rules . 86 3.2.1 Convergence Analysis . 89 3.2.2 Convergence Rates (for the Heuristic Discrepancy Rule) 92 3.3 Diagonal Operator Case Study . 94 3.3.1 Muckenhoupt Conditions . 97 4 Iterative Regularisation 108 4.1 Landweber Iteration for Linear Operators . 108 4.1.1 Heuristic Stopping Rules . 112 4.2 Landweber Iteration for Nonlinear Operators . 122 4 CONTENTS 5 4.2.1 Heuristic Parameter Choice Rules . 123 II Numerics 125 5 Semi-Heuristic Rules 127 5.1 Gaußian Operator Noise Perturbation . 128 5.1.1 Tomography Operator Perturbed by Gaußian Operator. 128 5.2 Smooth Operator Perturbation . 129 5.2.1 Fredholm Integral Operator Perturbed by Heat Operator129 5.2.2 Blur Operator Perturbed by Tomography Operator . 130 5.3 Summary . 131 6 Heuristic Rules for Convex Regularisation 136 6.1 `1 Regularisation . 137 3 6.2 ` 2 Regularisation . 138 6.3 `3 Regularisation . 139 6.4 TV Regularisation . 139 6.5 Summary . 140 7 The Simple L-curve Rules for Linear and Convex Tikhonov Regularisation 144 7.1 Linear Tikhonov Regularisation . 145 7.1.1 Diagonal Operator . 145 7.1.2 Examples from IR Tools . 146 7.2 Convex Tikhonov Regularisation . 148 7.2.1 `1 Regularisation . 148 3 7.3 ` 2 Regularisation . 149 7.4 TV Regularisation . 150 7.5 Summary . 151 8 Heuristic Rules for Nonlinear Landweber Iteration 152 8.1 Test problems . 154 8.1.1 Nonlinear Hammerstein Operator . 154 8.1.2 Auto-Convolution . 154 8.1.3 Summary . 157 III Future Scope 158 9 Future Scope 159 9.1 Convex Heuristic Regularisation . 159 9.2 Heuristic Blind Kernel Deconvolution . 159 9.2.1 Deconvolution . 160 9.2.2 Semi-Blind Deconvolution . 160 9.3 Meta-Heuristics . 162 A Functional Calculus 177 B Convex Analysis 181 6 Chapter 1 Introduction Typically in the \real world", we have problems in which we would like to extract information from given data, e.g., acoustic sound waves and X-ray sinograms among other examples. In particular, an acoustic sound wave recorded on the surface of the Earth contains information regarding the sub- surface, and X-rays contain information on the density of the material which they pass through. In order to recover this information, one must, in ef- fect, reverse the aforementioned processes, i.e., solve the inverse problem. In the theory of inverse problems, this is usually mathematically formalised in operator theoretic terms. That is, we generally consider an equation of the form Ax = y; (1.1) in which A : X ! Y is a continuous operator mapping between two vector spaces, called the \forward operator". The objective is then to invert the forward operator and to thus recover the solution x from measured data y. Generally speaking, the data we measure is considered corrupted to reflect, for instance, real-world machine error, and what we consider in fact is a perturbation of the data, yδ = y + e (where e may be very small), which we call noisy data, and then naturally y is called exact data. One should mention that the noise model, i.e., e may be deterministic or stochastic, although in this thesis, we will limit ourselves to the deterministic framework for ill-posed problems.