CODE GENERATION FOR EMBEDDED CONVEX OPTIMIZATION A DISSERTATION SUBMITTED TO THE DEPARTMENT OF ELECTRICAL ENGINEERING AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Jacob Elliot Mattingley February 2011 © 2011 by Jacob Elliot Mattingley. All Rights Reserved. Re-distributed by Stanford University under license with the author. This dissertation is online at: http://purl.stanford.edu/qd834ss0073 ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Stephen Boyd, Primary Adviser I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Michael Saunders I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Yinyu Ye Approved for the Stanford University Committee on Graduate Studies. Patricia J. Gumport, Vice Provost Graduate Education This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file in University Archives. iii Abstract Convex optimization is widely used, in many fields, but is nearly always constrained to problems solved in a few minutes or seconds, and even then, nearly always with a human in the loop. The advent of parser-solvers has made convex optimization simpler and more accessible, and greatly increased the number of people using convex optimization. Most current applications, however, are for the design of systems or analysis of data. It is possible to use convex optimization for real-time or embedded applications, where the optimization solver is a part of a larger system. Here, the optimization algorithm must find solutions much faster than a generic solver, and often has a hard, real-time deadline. Use in embedded applications additionally means that the solver cannot fail, and must be robust even in the presence of relatively poor quality data. For ease of embedding, the solver should be simple, and have minimal dependencies on external libraries. Convex optimization has been successfully applied in such settings in the past. However, they have usually necessitated a custom, hand-written solver. This requires signficant time and expertise, and has been a major factor preventing the adoption of convex optimization in embedded applications. This work describes the implementation and use of a prototype code generator for convex optimization, CVXGEN, that creates high-speed solvers automatically. Using the principles of disciplined convex programming, CVXGEN allows the user to describe an op- timization problem in a convenient, high-level language, then receive code for compilation into an extremely fast, robust, embeddable solver. iv Acknowledgments It is with gratitude that I look back over my time at Stanford and think of the many people who have helped me get this far. A few weeks after arriving, I was captivated by Professor Stephen Boyd’s teaching, and realized how much I could learn from him. When the opportunity arose, it was an easy choice to join his group. His teaching, assistance, and mentoring made the graduate school experience extremely satisfying. I do not think I could have had a better advisor. Other professors have helped as well; in particular, I would like to thank Professors Antony Fraser-Smith, Yinyu Ye, Sanjay Lall, Michael Saunders and Lieven Vandenberghe. Each has played an important part. The various members of my research group have been inspiring, helpful and supportive: Michael, Alessandro, Almir, Siddharth, Joëlle, Kwangmoo, Argyris, Yang, Ekine, Brendan, Eric, Matt, and Arezou. I have made many close friends at Stanford and look forward to keeping those rela- tionships for life. And finally, I would like to thank my family for their love and support: Barry, Lorelei, Ruth, and Daniel. This thesis is dedicated to them. v Contents Abstract iv Acknowledgments v 1 Overview 1 2 CodeGenerationforReal-TimeConvexOptimization 4 2.1 Introduction.................................... 4 2.2 Solvers and specification languages . 10 2.3 Examples ..................................... 16 2.4 Algorithmconsiderations. 26 2.5 Summary . 29 3 Implementation of a Convex Optimization Code Generator 30 3.1 Introduction.................................... 30 3.2 CVXGENexample ................................ 32 3.3 Problemspecification............................... 35 3.4 UsingCVXGEN ................................. 38 3.5 Implementation.................................. 41 3.6 Numericalexamples ............................... 52 3.7 Conclusion..................................... 58 4 Code Generation for Signal Processing 60 4.1 Introduction.................................... 60 4.2 Codegeneration ................................. 64 4.3 Linearizing pre-equalization . 65 4.4 RobustKalmanfiltering ............................. 68 vi 4.5 Conclusion..................................... 72 5 CodeGenerationforRecedingHorizonControl 74 5.1 Introduction.................................... 74 5.2 FormulatingRHCproblems . 76 5.3 Examples ..................................... 80 5.4 CVXGEN performance . 98 5.5 Conclusion..................................... 99 Bibliography 101 vii Chapter 1 Overview We start in Chapter 2 by introducing the use of convex optimization in real-time embedded systems, in areas such as signal processing, automatic control, real-time estimation, real- time resource allocation and decision making, and fast automated trading. By ‘embedded’ we mean that the optimization algorithm is part of a larger, fully automated system that executes automatically with newly arriving data or changing conditions, and without any human intervention or action. By ‘real-time’ we mean that the optimization algorithm executes much faster than a typical or generic method with a human in the loop, in times measured in milliseconds or microseconds for small and medium size problems, and (a few) seconds for larger problems. In real-time embedded convex optimization the same optimization problem is solved many times, with different data, often with a hard real- time deadline. In Chapter 2 we introduce the idea of using an automatic code generation system for real-time embedded convex optimization. Such a system scans a description of the problem family, and performs much of the analysis and optimization of the algorithm, such as choosing variable orderings used with sparse factorizations and determining stor- age structures, at code generation time. Compiling the generated source code yields an extremely efficient custom solver for the problem family. This chapter is based on [MB10]. Chapter 3 looks at a particular example of an automatic code generator, CVXGEN, which is a prototype code generator we developed to implement and test these ideas. CVX- GEN is a software tool that takes a high-level description of a convex optimization problem family and automatically generates custom C code that compiles into a reliable, high-speed solver for the problem family. The current implementation targets problem families that can be transformed, using disciplined convex programming techniques, to convex quadratic programs of modest size. CVXGEN generates simple, flat, library-free code suitable for 1 CHAPTER 1. OVERVIEW 2 embedding in real-time applications. The generated code is almost branch free, so has highly predictable run-time behavior. The combination of regularization (both static and dynamic) and iterative refinement in the search direction computation yields reliable per- formance, even with poor quality data. We describe how CVXGEN is implemented, and give some results on the speed and reliability of the automatically generated solvers. This chapter is based on [MB11]. Chapter 4 discusses signal processing applications of embedded convex optimization. Convex optimization has been used in signal processing for a long time, to choose coeffi- cients for use in fast (linear) algorithms, such as in filter or array design; more recently, it has been used to carry out (nonlinear) processing on the signal itself. Examples of the lat- ter case include total variation de-noising, compressed sensing, fault detection, and image classification. In both scenarios, the optimization is carried out on timescales of seconds or minutes, and without strict time constraints. Convex optimization has traditionally been considered computationally expensive, so its use has been limited to applications where plenty of time is available. Such restrictions are no longer justified. The combination of dramatically increased computing power, modern algorithms, and new coding approaches has delivered an enormous speed increase, which makes it possible to solve modest-sized convex optimization problems on microsecond or millisecond timescales, and with strict deadlines. This enables real-time convex optimization in signal processing. This chapter is based on [MB09]. The final chapter, Chapter 5, considers another important early application for code generation, receding horizon control (RHC), also known as model predictive control (MPC). RHC is a general-purpose control scheme that involves repeatedly solving a constrained optimization problem, using predictions of future costs, disturbances, and constraints over