Structured Matrix Nearness Problems: Theory and Algorithms A thesis submitted to the University of Manchester for the degree of Doctor of Philosophy in the Faculty of Engineering and Physical Sciences 2012 Ruediger Borsdorf School of Mathematics Contents Abstract 9 Declaration 10 Copyright Statement 11 Publications 12 Acknowledgements 13 1 Introduction 15 1.1 EmergenceofMatrixNearnessProblems . 15 1.1.1 ModellingofComplexSystems . 15 1.1.2 DeterminationofModelParameters. 15 1.1.3 StructuredMatrixNearnessProblems. 16 1.1.4 Examples Where Matrix Nearness Problems Occur . 16 1.2 GoalofThesis............................... 17 1.3 The Q-WeightedNorm .......................... 18 1.4 Structured Matrix Nearness Problems for ClosedConvex. 20 X 1.4.1 TheProblemanditsProperties . 20 1.4.2 AlternatingProjectionsMethod . 20 1.5 Set of Linearly Structured Matrices .................. 21 L 1.5.1 Definition of .......................... 21 L 1.5.2 Projection onto ......................... 22 L 1.6 Set of Positive Semidefinite Matrices + ................ 23 Sn 1.6.1 Definition of + .......................... 23 Sn 1.6.2 Projection onto + ........................ 23 Sn 1.7 Applications................................ 25 1.7.1 CorrelationMatrices . 25 1.7.2 ToeplitzandHankelMatrices . 25 1.7.3 CirculantMatrices . 26 2 1.7.4 SylvesterMatrices . 27 1.8 OutlineofThesis ............................. 28 1.9 MainResearchContributions . 29 2 NearestMatriceswithFactorStructure 32 2.1 Introduction................................ 32 2.2 OneParameterProblem . 34 2.3 OneFactorProblem ........................... 37 2.4 k FactorProblem ............................. 41 2.5 NumericalMethods............................ 43 2.6 ComputationalExperiments . 46 2.6.1 Test Results for k =1 ...................... 49 2.6.2 Choice of Starting Matrix, and Performance as k Varies. 51 2.6.3 Test Results for k > 1 ...................... 53 2.7 Conclusions ................................ 55 3 RiemannianGeometryandOptimization 56 3.1 Introduction................................ 56 3.1.1 Motivation for Optimizing over Manifolds . 56 3.1.2 Applications............................ 57 3.1.3 Outline .............................. 57 3.2 SmoothManifolds............................. 58 3.2.1 Definition ............................. 58 3.2.2 ExamplesofSmoothManifolds . 59 3.3 SmoothFunctionsandTangentSpaces . 59 3.3.1 SmoothFunctions. 60 3.3.2 TangentVectorsandSpaces . 60 3.4 EmbeddedSubmanifolds . 61 3.4.1 RecognizingEmbeddedSubmanifolds . 61 3.4.2 ManifoldsEmbeddedinEuclideanSpace . 62 3.5 QuotientManifolds ............................ 63 3.5.1 Definition ............................. 63 3.5.2 SmoothFunctions. 63 3.5.3 TangentSpace........................... 64 3.5.4 Quotient Manifolds Embedded in Euclidean Space . .. 64 3.6 RiemannianManifolds . 64 3.6.1 RiemannianMetricandDistance . 64 3.6.2 RiemannianSubmanifold. 65 3.6.3 RiemannianQuotientManifold . 65 3 3.7 GeometricObjects ............................ 66 3.7.1 TheGradient ........................... 66 3.7.2 Levi-CivitaConnection. 66 3.7.3 GeodesicsandRetractions . 68 3.7.4 TheRiemannianHessian. 69 3.7.5 VectorTransport ......................... 69 3.8 ExamplesofRiemannianManifolds . 71 3.8.1 TheStiefelManifold . 71 3.8.2 TheGrassmannianManifold . 73 3.9 OptimizationAlgorithms. 76 3.9.1 NonlinearConjugateGradientAlgorithm . 76 3.9.2 LimitedMemoryRBFGS. 79 3.10Conclusions ................................ 82 4 Two-Sided Optimization Problems 84 4.1 Introduction................................ 84 4.2 TheProblems............................... 85 4.2.1 Problem1............................. 85 4.2.2 Problem2............................. 85 4.3 OptimalityConditionsforProblem1 . 86 4.3.1 ConditionsforStationaryPoints. 86 4.3.2 AttainingOptimalFunctionValue. 87 4.4 StepstoOptimalSolutionofProblem1. 88 4.4.1 Construction of Arrowhead Matrix with Prescribed Eigenspec- trum................................ 89 4.4.2 ObtaininganOptimalSolution . 90 4.5 StepstoOptimalSolutionofProblem2. 93 4.5.1 Reformulation into a Convex Quadratic Programming . ... 94 4.5.2 Active-Set Method for Convex Quadratic Problems . 94 4.5.3 ApplyingActive-SetMethodtoProblem2 . 96 4.6 Optimizing Functions over Set of Optimal Solutions . ...... 101 4.6.1 Introduction............................ 101 4.6.2 Modified Constraint Set Forming Riemannian Manifold . 101 4.6.3 GeometricObjectsofthisManifold . 104 4.6.4 Optimization overWholeConstraintSet . 110 4.7 ComputationalExperiments . 115 4.7.1 TestProblem ........................... 116 4.7.2 NumericalMethods. 117 4 4.7.3 TestMatricesandStartingValues. 117 4.7.4 NumericalTests. 118 4.8 Conclusions ................................ 126 5 LowRankProblemofStructuredMatrices 128 5.1 Introduction................................ 128 5.1.1 TheProblem ........................... 128 5.1.2 Applications............................ 129 5.1.3 Outline .............................. 130 5.2 Algorithms Dealing with Any Linear Structure . 130 5.2.1 TheLiftandProjectionAlgorithm . 130 5.2.2 Transformation into a Structured Total Least Norm Problem . 131 5.2.3 Reformulating and Applying Geometric Optimization . 133 5.3 StepstoOurMethod........................... 136 5.3.1 Applying the Augmented Lagrangian Method . 138 5.3.2 Steps to Compute fµ,λ ...................... 139 5.3.3 Forming the Derivative of the Objective Function . 139 5.3.4 Convergence............................ 141 5.3.5 OurAlgorithm .......................... 144 5.4 ComputationalExperiments . 144 5.4.1 TestMatrices ........................... 146 5.4.2 NumericalMethods. 147 5.4.3 NumericalTests. 148 5.5 Conclusions ................................ 156 6 Conclusions and Future Work 158 List of Symbols 161 A Some Definitions 163 A.1 KroneckerProduct ............................ 163 A.1.1 Definition ............................. 163 A.1.2 Properties ............................. 163 A.2 Fr´echetDerivative. 164 Bibliography 165 Word count 65946 5 List of Tables 2.1 Summary of the methods, with final column indicating the available convergence results (see the text for details). .... 47 3 2.2 Results for the random one factor problems with tol = 10− . ..... 50 6 2.3 Results for the random one factor problems with tol = 10− . ..... 50 3 2.4 Results for the one factor problem for cor1399 with tol = 10− and 6 tol = 10− .................................. 50 3 2.5 Results for the random k factor problems with tol = 10− . ...... 54 6 2.6 Results for the random k factor problems with tol = 10− . ...... 54 4.1 Output for ALB for test matrices ldchem................ 121 4.2 Results for the randomly generated matrices Λ and D. ........ 122 4.3 Results for different methods to solve the linear system H2z2 = b2 in (4.35). ................................... 125 5.1 Performance of Algorithm 5.3.1 for different methods to solve (5.23) for test matrices uhankel and r = n 5................. 149 − 5.2 Results for Algorithm 5.3.1 for different methods to solve (5.23) for test matrices uhankel........................... 151 5.3 Results for test matrices of type uexample. .............. 152 5.4 Results for r = p 1 and test matrices urand.............. 153 − 6 List of Figures 2.1 Comparison of different starting values for matrices of type randneig: k against final objective function value (left) and time (right). .... 52 2.2 Comparison of different starting values for matrices of type expij: k against final objective function value (left) and time (right). ..... 53 4.1 Ratio of time spent on computing the projection to total time . 123 4.2 Comparisonoffunctionvalues . 124 4.3 Rank of grad c(Y ) ............................ 124 ∗ 5.1 1 A X 2 against the objective rank r................. 153 2 − ∗Q 5.2 Xssv against the objective rank r..................... 154 5.3 Computational time in seconds against the objective rank r. ..... 154 5.4 Norm of gradient against number of iterations in RBFGS algorithm. 155 7 List of Algorithms 3.9.1 Nonlinear Conjugate Gradient Algorithm on Riemannian manifold . 78 M 3.9.2 Algorithm to compute Hk(ξxk ) for the limited memory BFGS. 83 3.9.3 Limited memory BFGS algorithm for Riemannian manifolds...... 83 4.4.1Algorithm for computing the solution of (4.1). ...... 93 4.5.1 Active-set method for computing the solution of (4.14)......... 100 4.6.1 (ALB) This algorithm minimizes an arbitrary smooth function f over the set in(4.24). ............................ 112 C 5.3.1 This algorithm finds the nearest low rank linearly structured matrix to a given linearly structured matrix by minimizing (5.18). ...... 145 8 The University of Manchester Ruediger Borsdorf June 11, 2012 Doctor of Philosophy Structured Matrix Nearness Problems: Theory and Algorithms In many areas of science one often has a given matrix, representing for example a measured data set and is required to find a matrix that is closest in a suitable norm to the matrix and possesses additionally a structure, inherited from the model used or coming from the application. We call these problems structured matrix nearness problems. We look at three different groups of these problems that come from real applications, analyze the properties of the corresponding matrix structure, and propose algorithms to solve them efficiently. The first part of this thesis concerns the nearness problem of finding the nearest T T k factor correlation matrix C(X) = diag(In XX )+ XX to a given symmetric matrix, subject to natural nonlinear constraints− on the elements of the n k matrix X, where distance is measured in the Frobenius norm. Such problems×