Reactive Design Patterns

ROLAND KUHN

WITH BRIAN HANAFEE AND JAMIE ALLEN

FOREWORD BY JONAS BONÉR

MANNING SHELTER ISLAND For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: [email protected]

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

Manning Publications Co. Development editor: Jennifer Stout 20 Baldwin Road Technical development editor: Brian Hanafee PO Box 761 Project editors: Tiffany Taylor and Janet Vail Shelter Island, NY 11964 Line editor: Ben Kovitz Copyeditor: Tiffany Taylor Proofreader: Katie Tennant Technical proofreader: Thomas Lockney Typesetter: Dottie Marsico Cover designer: Leslie Haimes

ISBN 9781617291807 Printed in the United States of America 1 2 3 4 5 6 70 8 9 10 – EBM – 22 21 20 19 18 17 To my children — Roland

brief contents

PART 1 INTRODUCTION...... 1 1 ■ Why Reactive? 3 2 ■ A walk-through of the Reactive Manifesto 12 3 ■ Tools of the trade 39

PART 2 THE PHILOSOPHY IN A NUTSHELL ...... 65 4 ■ Message passing 67 5 ■ Location transparency 81 6 ■ Divide and conquer 91 7 ■ Principled failure handling 100 8 ■ Delimited consistency 105 9 ■ Nondeterminism by need 113 10 ■ Message flow 120

PART 3 PATTERNS ...... 125 11 ■ Testing reactive applications 127 12 ■ Fault tolerance and recovery patterns 162 13 ■ Replication patterns 184 14 ■ Resource-management patterns 220 15 ■ Message flow patterns 255 16 ■ Flow control patterns 294 17 ■ State management and persistence patterns 311

contents

foreword xv preface xvii acknowledgments xix about this book xxi about the authors xxiv

PART 1 INTRODUCTION...... 1 Why Reactive? 3 1 1.1 The anatomy of a Reactive application 4 1.2 Coping with load 5 1.3 Coping with failures 7 1.4 Making the system responsive 8 1.5 Avoiding the ball of mud 10 1.6 Integrating nonreactive components 10 1.7 Summary 11

A walk-through of the Reactive Manifesto 12 2 2.1 Reacting to users 12

Understanding the traditional approach 13 ■ Analyzing latency with a shared resource 15 ■ Limiting maximum latency with a queue 16

vii viii CONTENTS

2.2 Exploiting parallelism 18

Reducing latency via parallelization 18 ■ Improving parallelism with composable Futures 20 ■ Paying for the serial illusion 21 2.3 The limits of parallel execution 23

Amdahl’s Law 23 ■ Universal Scalability Law 24 2.4 Reacting to failure 26

Compartmentalization and bulkheading 28 ■ Using circuit breakers 29 ■ Supervision 30 2.5 Losing strong consistency 31

ACID 2.0 33 ■ Accepting updates 34 2.6 The need for Reactive design patterns 35

Managing complexity 36 ■ Bringing programming models closer to the real world 37 2.7 Summary 38

Tools of the trade 39 3 3.1 Early Reactive solutions 39 3.2 Functional programming 41

Immutability 41 ■ Referential transparency 44 ■ Side effects 45 ■ Functions as first-class citizens 46 3.3 Responsiveness to users 47 Prioritizing the three performance characteristics 47 3.4 Existing support for Reactive design 48

Green threads 48 ■ Event loops 49 ■ Communicating Sequential Processes 50 ■ Futures and promises 52 Reactive Extensions 57 ■ The Actor model 58 Summary 62

PART 2 THE PHILOSOPHY IN A NUTSHELL ...... 65

Message passing 67 4 4.1 Messages 67 4.2 Vertical scalability 68 4.3 Event-based vs. message-based 69 4.4 Synchronous vs. asynchronous 71 4.5 Flow control 73 CONTENTS ix

4.6 Delivery guarantees 76 4.7 Events as messages 78 4.8 Synchronous message passing 80 4.9 Summary 80

Location transparency 81 5 5.1 What is location transparency? 81 5.2 The fallacy of transparent remoting 82 5.3 Explicit message passing to the rescue 83 5.4 Optimization of local message passing 84 5.5 Message loss 85 5.6 Horizontal scalability 87 5.7 Location transparency makes testing simpler 87 5.8 Dynamic composition 88 5.9 Summary 90

Divide and conquer 91 6 6.1 Hierarchical problem decomposition 92 Defining the hierarchy 92 6.2 Dependencies vs. descendant modules 94 Avoiding the matrix 95 6.3 Building your own big corporation 96 6.4 Advantages of specification and testing 97 6.5 Horizontal and vertical scalability 98 6.6 Summary 99

Principled failure handling 100 7 7.1 Ownership means commitment 100 7.2 Ownership implies lifecycle control 102 7.3 Resilience on all levels 104 7.4 Summary 104

Delimited consistency 105 8 8.1 Encapsulated modules to the rescue 106 8.2 Grouping data and behavior according to transaction boundaries 107 x CONTENTS

8.3 Modeling workflows across transactional boundaries 107 8.4 Unit of failure = unit of consistency 108 8.5 Segregating responsibilities 109 8.6 Persisting isolated scopes of consistency 111 8.7 Summary 112

Nondeterminism by need 113 9 9.1 Logic programming and declarative data flow 113 9.2 Functional reactive programming 115 9.3 Sharing nothing simplifies concurrency 116 9.4 Shared-state concurrency 117 9.5 So, what should we do? 117 9.6 Summary 119

Message flow 120 10 10.1 Pushing data forward 120 10.2 Modeling the processes of your domain 122 10.3 Identifying resilience limitations 122 10.4 Estimating rates and deployment scale 123 10.5 Planning for flow control 124 10.6 Summary 124

PART 3 PATTERNS ...... 125 Testing reactive applications 127 11 11.1 How to test 127

Unit tests 128 ■ Component tests 129 ■ String tests 129 Integration tests 129 ■ User-acceptance tests 130 ■ Black-box vs. white-box tests 130 11.2 Test environment 131 11.3 Testing asynchronously 132

Providing blocking message receivers 133 ■ The crux of choosing timeouts 135 ■ Asserting the absence of a message 141 Providing synchronous execution engines 142 ■ Asynchronous assertions 144 ■ Fully asynchronous tests 145 ■ Asserting the absence of asynchronous errors 147 CONTENTS xi

11.4 Testing nondeterministic systems 150

The trouble with execution schedules 151 ■ Testing distributed components 151 ■ Mocking Actors 152 ■ Distributed components 153 11.5 Testing elasticity 154 11.6 Testing resilience 154

Application resilience 155 ■ Infrastructure resilience 158 11.7 Testing responsiveness 160 11.8 Summary 161

Fault tolerance and recovery patterns 162 12 12.1 The Simple Component pattern 162

The problem setting 163 ■ Applying the pattern 163 The pattern, revisited 165 ■ Applicability 166 12.2 The Error Kernel pattern 166

The problem setting 167 ■ Applying the pattern 167 The pattern, revisited 170 ■ Applicability 171 12.3 The Let-It-Crash pattern 171

The problem setting 172 ■ Applying the pattern 172 The pattern, revisited 173 ■ Implementation considerations 174 ■ Corollary: the Heartbeat pattern 175 Corollary: the Proactive Failure Signal pattern 176 12.4 The Circuit Breaker pattern 177

The problem setting 177 ■ Applying the pattern 178 The pattern, revisited 181 ■ Applicability 182 12.5 Summary 182

Replication patterns 184 13 13.1 The Active–Passive Replication pattern 184

The problem setting 185 ■ Applying the pattern 186 The pattern, revisited 196 ■ Applicability 197 13.2 Multiple-Master Replication patterns 197

Consensus-based replication 198 ■ Replication with conflict detection and resolution 201 ■ Conflict-free replicated data types 203 13.3 The Active–Active Replication pattern 210

The problem setting 211 ■ Applying the pattern 211 xii CONTENTS

The pattern, revisited 217 ■ The relation to virtual synchrony 218 13.4 Summary 219

Resource-management patterns 220 14 14.1 The Resource Encapsulation pattern 221

The problem setting 221 ■ Applying the pattern 221 The pattern, revisited 227 ■ Applicability 228 14.2 The Resource Loan pattern 229

The problem setting 229 ■ Applying the pattern 230 The pattern, revisited 232 ■ Applicability 233 Implementation considerations 233 ■ Variant: using the Resource Loan pattern for partial exposure 234 14.3 The Complex Command pattern 234

The problem setting 235 ■ Applying the pattern 236 The pattern, revisited 243 ■ Applicability 243 14.4 The Resource Pool pattern 244

The problem setting 244 ■ Applying the pattern 245 The pattern, revisited 247 ■ Implementation considerations 248 14.5 Patterns for managed blocking 248

The problem setting 249 ■ Applying the pattern 249 The pattern, revisited 251 ■ Applicability 253 14.6 Summary 253

Message flow patterns 255 15 15.1 The Request–Response pattern 255

The problem setting 256 ■ Applying the pattern 257 Common instances of the pattern 258 ■ The pattern, revisited 263 ■ Applicability 264 15.2 The Self-Contained Message pattern 265

The problem setting 265 ■ Applying the pattern 266 The pattern, revisited 268 ■ Applicability 268 15.3 The Ask pattern 269

The problem setting 269 ■ Applying the pattern 270 The pattern, revisited 273 ■ Applicability 274 15.4 The Forward Flow pattern 275

The problem setting 275 ■ Applying the pattern 275 The pattern, revisited 276 ■ Applicability 276 CONTENTS xiii

15.5 The Aggregator pattern 277

The problem setting 277 ■ Applying the pattern 277 The pattern, revisited 281 ■ Applicability 281 15.6 The Saga pattern 282

The problem setting 282 ■ Applying the pattern 283 The pattern, revisited 284 ■ Applicability 286 15.7 The Business Handshake pattern (a.k.a. Reliable Delivery pattern) 286

The problem setting 287 ■ Applying the pattern 287 The pattern, revisited 292 ■ Applicability 292 15.8 Summary 293

Flow control patterns 294 16 16.1 The Pull pattern 294

The problem setting 295 ■ Applying the pattern 295 The pattern, revisited 297 ■ Applicability 298 16.2 The Managed Queue pattern 298

The problem setting 298 ■ Applying the pattern 299 The pattern, revisited 300 ■ Applicability 301 16.3 The Drop pattern 301

The problem setting 301 ■ Applying the pattern 302 The pattern, revisited 303 ■ Applicability 306 16.4 The Throttling pattern 306

The problem setting 307 ■ Applying the pattern 307 The pattern, revisited 309 16.5 Summary 310

State management and persistence patterns 311 17 17.1 The Domain Object pattern 312

The problem setting 312 ■ Applying the pattern 313 The pattern, revisited 315 17.2 The Sharding pattern 316

The problem setting 316 ■ Applying the pattern 316 The pattern, revisited 318 ■ Important caveat 319 17.3 The Event-Sourcing pattern 319

The problem setting 319 ■ Applying the pattern 320 The pattern, revisited 321 ■ Applicability 322 xiv CONTENTS

17.4 The Event Stream pattern 322

The problem setting 323 ■ Applying the pattern 323 The pattern, revisited 325 ■ Applicability 325 17.5 Summary 326

appendix A Diagramming Reactive systems 327 appendix B An illustrated example 329 appendix C The Reactive Manifesto 343

index 351 foreword

I’m grateful that Roland has taken the time to write this foundational book, and I can’t think of anyone more capable of pulling it off. Roland is an unusually clear and deep thinker; he coauthored the Reactive Manifesto, has been the technical lead for the Akka project for several years, has coauthored and taught the very popular Cour- sera course on Reactive programming and design, and is the best technical writer I have met. Clearly, I’m very excited about this book. It outlines what Reactive architecture/design is all about, and does an excellent job explaining it from first principles in a practical context. Additionally, it is a catalog of patterns that explains the bigger picture, how to think about system design, and how it is all connected—much like what Martin Fowler’s Patterns of Enterprise Application Architecture did 15 years ago. During my professional life, I have seen the immense benefits of resilient, loosely coupled, message-driven systems firsthand, especially when compared with more- traditional approaches that propose to hide the nature of distributed systems. In 2013, I had the idea of formalizing the experiences and lessons learned: the Reactive Mani- festo was born. It started out as a set of rough notes that I remember presenting to the company at one of Typesafe’s (now Lightbend) internal technical meetups. Coinci- dentally, this meetup was collocated with the Scala Days New York conference, where Roland, Martin Odersky, and Erik Meijer shot their bad, and unintentionally quite funny, promotion video of their Coursera course on Reactive programming. The story around the Reactive principles resonated with the other engineers and was published in July of 2013. Since then, the Manifesto has been receiving a lot of great feedback from the community. It was rewritten and vastly improved by Roland, Martin Thomp- son, Dave Farley, and myself, leading up to version 2.0 published in September 2014.

xv xvi FOREWORD

By the end of 2016, it had been signed by more than 17,000 people. During this time, we have seen Reactive progress from a virtually unacknowledged technique used only by fringe projects within a select few corporations to a part of the overall platform strategy of numerous big players in many different fields, including middleware, financial services, retail, social media, betting/gaming, and so on. The Reactive Manifesto defines “Reactive Systems” as a set of architectural design principles that are geared toward meeting the demands that systems face—today and tomorrow. These principles are most definitely not new; they can be traced back to the ’70s and ’80s and the seminal work by Jim Gray and Pat Helland on the Tandem System, as well as Joe Armstrong and Robert Virding on Erlang. However, these pio- neers were ahead of their time, and it was not until the past five years that the technology industry was forced to rethink current best practices for enterprise system development and learned to apply the hard-won knowledge of the Reactive principles to today’s world of multicore architectures, Cloud Computing, and the Internet of Things. By now, the Reactive principles have had a big impact on the industry, and as with many successful ideas, they get overloaded and reinterpreted. This is not a bad thing; ideas need to evolve to stay relevant. However, this can also cause confusion and lead to dilution of the original intent. One example is the unfortunate emerging miscon- ception that Reactive is nothing but programming in an asynchronous and nonblocking style using callbacks or stream-oriented combinators—techniques that are aptly classified as Reactive Programming. Concentrating on this aspect alone means missing out on many of the benefits of the Reactive principles. It is the contribution of this book to take a much larger perspective—a systems view—moving the focus from how individual components function in isolation to the design of collaborative, resilient, and elastic systems: Reactive systems. This future classic belongs on the shelf of every professional programmer, right next to GoF1 and Domain-Driven Design.2!Enjoy the ride—I certainly did

JONAS BONÉR CTO AND FOUNDER OF LIGHTBEND CREATOR OF AKKA

1 Design Patterns: Elements of Reusable Object-Oriented Software by Gamma, Helm, Johnson, and Vlissides (Addison- Wesley, 1995). 2 Domain-Driven Design by Eric Evans (Addison-Wesley, 2004). preface

Even before I had officially joined the Akka team, Mike Stephens from Manning tried to convince me to write a book on Akka. I was tempted to say yes, but in the context of an impending change of jobs and countries, my wife brought me to my senses: such a project would be too much to handle. The idea of writing a book stuck in my head, though. Three years later—after the Reactive Manifesto had been published—Martin Odersky, Erik Meijer, and I taught the course Principles of Reactive Programming on the Coursera platform, reaching more than 120,000 students in two iterations. The idea for that course had been born at a Typesafe engineering meeting where I sug- gested to Martin that we should nurture the blossoming movement of Reactive programming by demonstrating how to use these tools effectively while avoiding the pitfalls—my own experience answering questions on the Akka mailing list had given me a good idea of the topics people commonly struggled with. A video course is a wonderful way of reaching a large number of students, interacting with them on the discussion forums, and in general improving the lives of others. Unfortunately, the discussion of the subject is necessarily limited in its depth and breadth by the format: only so much can be shown in seven weekly lectures. Therefore, I still longed for formalizing and passing on my knowledge about Reactive systems by writing a book. It would have been straightforward to write about Akka, but I felt that if I wrote a book, its scope should be wider than that. I love working on Akka—it has literally changed the course of my life—but Akka is merely a tool for expressing distributed and highly reliable systems, and it is not the only tool needed in this regard. Thus began the journey toward the work you are holding in your hands right now. It was a daunting task, and I knew that I would need help. Luckily, Jamie was just

xvii xviii PREFACE

about to finish Effective Akka 3 and was immediately on board. Neither of us had the luxury of writing during daytime; consequently, the book started out slow and kept lagging behind the plan. Instead of having three chapters ready to enter the early access program during the first iteration of the course Principles of Reactive Program- ming, we could only announce it several months later. It is astonishing how much detail one finds missing when starting out from the viewpoint that the contents are basically already known and just need to be transferred into the computer. Over time, Jamie got even busier with his day job, until he had to stop contributing entirely. Later, Brian joined the project as Manning’s technical development editor, and it soon became clear that he could not only make very good suggestions but also implement them. We made it official by signing him up as a coauthor, and then Brian helped me push the manuscript over the finish line. This book contains not only advice on when and how to use the tools of Reactive programming, but also the reasoning behind the advice, so that you may adapt it to different requirements and new applications. I hope that it will inspire you to learn more and to explore the wonderful world of Reactive systems.

ROLAND KUHN

3 Effective Akka by Jamie Allen (O’Reillly Media, 2013). acknowledgments

ROLAND KUHN My first thanks go to Jamie, without whom I would not have dared take on this project. But my deepest gratitude is to Jonas Bonér, who created Akka, entrusted Akka to my care for many years, and supported me on every step along the way. I am also deeply thankful to Viktor Klang for countless rigorous discussions about all topics of life (and distributed systems) but, more importantly, for teaching me how to lead by example and how important it is to not let the devil over the bridge. Jonas, Viktor, and Patrik Nordwall also deserve special thanks for covering my duties as Akka Tech Lead while I took a mini-sabbatical of three months to work intensely on this book. I greatly appreciate Brian and Jamie stepping up and shouldering part of the tremendous weight of such a project: it is gratifying and motivating to work alongside such trusted companions. For helpful reviews of the early manuscript, I would like to thank Sean Walsh and Duncan DeVore, as well as Bert Bates, who helped shape the overall arrangement of how the patterns are presented. I also thank Endre Varga, who spent considerable effort developing the KVStore exercise for Principles of Reactive Programming that forms the basis for the state replication code samples used in chapter 13. Thanks also go to Pablo Medina for helping me with the CKite example code in section 13.2, and to Thomas Lockney, technical proofreader, who kept a sharp eye out for errors. The following peer reviewers gave generously of their time: Joel Kotarski, Valentine Sinit- syn, Mark Elston, Miguel Eduardo Gil Biraud, William E. Wheeler, Jonathan Freeman, Franco Bulgarelli, Bryan Gilbert, Carlos Curotto, Andy Hicks, William Chan, Jacek Sokulski, Dr. Christian Bridge-Harrington, Satadru Roy, Richard Jepps, Sorbo Bagchi, NenkoTabakov, Martin Anlauf, Kolja Dummann, Gordon Fische, Sebastien Boisver,

xix xx ACKNOWLEDGMENTS

and Henrik Løvborg. I am grateful to the Akka community for being such a welcoming and fertile place for developing our understanding of distributed systems. I would like to thank the team at Manning who made this book possible, especially Mike Stephens for nagging until I gave in, Jenny Stout for urging me to make progress, and Candace Gillhoolley from marketing. I would like to distinguish Ben Kovitz as an extremely careful and thorough copy editor, and I thank Tiffany Taylor for finding even more redundant words to be removed from the final text, as well as Katie Tennant for identifying and fixing unclear passages. Finally, in the name of all readers, I extend my utmost appreciation and love to my wife, Alex. You endured my countless hours of spiritual absence with great compassion.

JAMIE ALLEN I wish to thank my wife, Yeon, and my three children, Sophie, Layla, and James. I am also grateful to Roland for allowing me to participate in this project, and to Brian for pushing the project over the finish line and contributing his expertise.

BRIAN HANAFEE I thank my wife, Patty, for supporting me, always, and my daughters, Yvonne and Barbara, for helping me with Doctor Who history and sometimes pretend- ing my jokes are funny. Thank you, Susan Conant and Bert Bates, for getting me started and teaching me how to edit and teach in book form. Finally, thank you, Roland and Jamie, for showing me Reactive principles and welcoming me into this project. about this book

This book is intended to be a comprehensive guide to understanding and designing Reactive systems. Therefore, it includes not only an annotated version of the Reactive Manifesto, but also the reasoning that led to its inception. The main part of the book is a selection of design patterns that implement many facets of Reactive system design, with pointers toward deeper literature resources for further study. While the presented patterns form a cohesive whole, the list is not exhaustive—it cannot be—but the included background knowledge will enable the reader to identify, distill, and curate new patterns as the need arises. Whom this book is for This book was written for everyone who may want to implement Reactive systems: It covers the architecture of such systems as well as the philosophy behind it, giving architects an overview of the characteristics of Reactive applications and their components and discussing the applicability of the patterns. Practitioners will benefit from a detailed discussion of the scenario solved by each pattern, the steps to take in applying it—illustrated with complete source code—as well as a guide to transfer and adapt the pattern to different cases. Learners wishing to deepen their knowledge, for example, after viewing the course material of Principles of Reactive Programming, will be delighted to read about the thought processes behind the Reactive principles and to follow the literature references for further study. This book does not require prior knowledge of Reactive systems; it builds upon famil- iarity with software development in general and refers to some experience with the

xxi xxii ABOUT THIS BOOK

difficulties arising from distributed systems. For some parts, a basic understanding of functional programming is helpful (in the sense of programming with immutable values and pure functions), but category theory does not feature in this book. How to read this book The contents of this book are arranged such that it lends itself well to being read as a story, cover to cover, developing from an introductory example and an overview of the Reactive Manifesto and the Reactive toolbox, continuing with the philosophy behind Reactive principles, and culminating in the description of patterns covering the different aspects of designing a Reactive system. This journey covers a lot of ground and the text contains references to additional background information. Reading it in one go will leave you with an intuition of the scope of the book and what information is found where, but it will typically only be the entry point for further study; you will return for the extraction of deeper insights while applying the acquired knowledge in projects of your own. If you are already familiar with the challenges of Reactive systems, you may skip the first chapter, and you will likely skim chapter 3 on the tools of the trade because you have already worked with most of those. The impatient will be tempted to start reading the patterns in part 3, but it is recommended to take a look at part 2 first: the pattern descriptions frequently refer to the explanations and background knowledge of this more theoretical part that form the basis on which the patterns have been developed. It is expected that you will return to the more philosophical chapters—especially chapters 8 and 9—after having gained more experience with the design and implementation of Reactive systems; don’t worry if these discussions do not immediately become fully transparent upon first reading. Conventions Due to the overloading of the English term “future” for a programming concept that deviates significantly from the original meaning, all uses of the word referring to the programming concept appear capitalized as Future, even when not appearing in code font. The situation is slightly different for the term “actor,” which in plain English refers to a person on stage as well as a participant in an action or process. This term appears capitalized only when referring specifically to the Actor model, or when the name of the Actor trait appears in code font. Source code for the examples All source code for the examples used in this book are available for download on GitHub here: https://github.com/ReactiveDesignPatterns/CodeSamples/. GitHub also offers facilities for raising issues with the samples or discussing them; please make use of them. You are also welcome to open pull requests with improvements; this way, all future readers will benefit from your thoughtfulness and experience. ABOUT THIS BOOK xxiii

Most of the samples are written in Java or Scala and use sbt for the build definition; please refer to www.scala-sbt.org/ for detailed documentation. A Java development kit supporting Java 8 will be required to build and run the samples. Other online resources An overview of the presented patterns as well as further material is available at www.reactivedesignpatterns.org/. In addition, purchase of Reactive Design Patterns includes free access to a private web forum run by Manning Publications where you can make comments about the book, ask technical questions, and receive help from the lead author and from other users. To access the forum and subscribe to it, point your web browser to www.manning.com/books/reactive-design-patterns. This page provides information on how to get on the forum once you are registered, what kind of help is available, and the rules of conduct on the forum. Manning’s commitment to our readers is to provide a venue where a meaningful dialog between individual readers and between readers and authors can take place. It is not a commitment to any specific amount of participation on the part of the authors, whose contribution to the AO forum remains voluntary (and unpaid). We suggest you try asking them some challenging questions lest their interest stray! The Author Online forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print. about the authors

Dr. Roland Kuhn studied physics at the Technische Universität München and obtained a doctorate with a dissertation on measurements of the gluon spin structure of the nucleon at a high-energy particle physics experiment at CERN (Geneva, Switzer- land). This entailed usage and implementation of large computing clusters and fast data processing networks, which laid the foundation for his thorough understanding of distributed computing. Afterward, he worked for four years at the German space operations center, building control centers and ground infrastructure for military satellite missions, before joining the Akka team at Lightbend (then called Typesafe), which he led from November 2012 to March 2016. During this time he co-taught the course Principles of Reactive Programming on the Coursera platform together with Martin Odersky and Erik Meijer, a course that was visited by more than 120,000 students. Together with Jonas Bonér, he authored the first version of the Reactive Mani- festo, published in June 2013. Currently, Roland is CTO of Actyx, a Munich-based company he cofounded, bringing the benefits of modern Reactive systems to small and midsize manufacturing enterprises across Europe.

Brian Hanafee received his BS in EECS from the University of California, Berkeley. He is a Principal Systems Architect at Wells Fargo Bank, where he designs internet banking and payment systems and is a consistent advocate for raising the technology bar. Previously he was with Oracle, working on new and emerging products and systems for interactive television and for text processing. He sent his first email from a moving vehicle in 1994. Prior to that, Brian was an Associate at Booz, Allen & Hamilton and at Advanced Decision Systems, where he applied AI techniques to military planning

xxiv ABOUT THE AUTHORS xxv systems. He also wrote software for one of the first ejection-safe helmet-mounted display systems.

Jamie Allen is the Director of Engineering for the UCP project at Starbucks, an effort to redefine the digital experience for every customer across all of our operating models and locations. He is the author of Effective Akka (O’Reilly, 2013), and previously worked with Roland and Jonas at Typesafe/Lightbend for over four years. Jamie has been a Scala and actor developer since 2008, and has worked with numerous clients around the world to help them understand and adopt Reactive application approaches.

Part 1 Introduction

H ave you ever wondered how high-profile web applications are implemented? Social networks and huge retail sites must have some secret ingredient that makes them work quickly and reliably, but what is it? In this book, you will learn about the design principles and patterns behind such systems that never fail and are capable of serving the needs of billions of people. Although the systems you build may not have such ambitious requirements, the primary qualities are common: You want your application to work reliably, even though parts (hardware or software) may fail. You want it to keep working when you have more users to support, and you want to be able to add or remove resources to adapt its capacity to changing demand (capacity planning is hard to get right without a crystal ball). In chapter 1, we will sketch the development of an application that exhibits these qualities and more. We will illustrate the challenges you will encounter and present solutions based on a concrete example—a hypothetical implementation of the Gmail service—but we will do so in a technology-agnostic fashion. This use case sets the stage for the detailed discussion of the Reactive Mani- festo that follows in chapter 2. The manifesto is written in a concise, high-level form in order to concentrate on its essence: the combination of individually useful program characteristics into a cohesive whole that is larger than the sum of its parts. We will show this by breaking the high-level traits into smaller pieces and explaining how everything fits back together.

2 PART 1 Introduction

We will complete this part of the book in chapter 3 with a whirlwind tour through the tools of the trade: functional programming, Futures and Promises, Communicat- ing Sequential Processes (CSP), Observers and Observables (Reactive Extensions), and the Actor model.

Why Reactive?

We start from the desire to build a system that is responsive to users. This means the system should respond to user input in a timely fashion under all circumstances. Because any single computer can fail at any time, we need to distribute such a system over multiple computers. Adding this fundamental requirement for distribution makes us recognize the need for new architecture patterns (or to rediscover old ones). In the past, we developed methods that allowed us to retain the illusion of single-threaded local processing while having it magically executed on multiple cores or network nodes, but the gap between that illusion and reality is becoming prohibitively large.1 The solution is to make the distributed, concurrent nature of our applications explicit in the programming model, using it to our advantage. This book will teach you how to write systems that stay responsive in the face of partial outages, program failure, changing loads, and even bugs in the code. You will see that this requires adjustments to the way you think about and design your applications. Here are the four tenets of the Reactive Manifesto,2 which defines a common vocabulary and lays out the basic challenges that a modern computer system needs to meet: It must react to its users (responsive). It must react to failure and stay available (resilient). It must react to variable load conditions (elastic). It must react to inputs (message-driven).

1 For example, Java EE services allow us to transparently call remote services that are wired in automatically, possibly even including distributed database transactions. The possibility of network failure or remote service overload, and so on, is completely hidden, abstracted away, and consequently out of reach for developers to meaningfully take into account. 2 http://reactivemanifesto.org

3 4 CHAPTER 1 Why Reactive?

Value Responsive Maintainable Extensible

Means Elastic Resilient

Form Message-driven Figure 1.1 The structure of Reactive values

In addition, creating a system with these properties in mind will guide you toward better modularization, both of the runtime deployment and of the code itself. Therefore, we add two more attributes to the list of benefits: maintainability and extensibility. Another way to structure the attributes is shown in figure 1.1. In the following chapters, you will learn about the reasoning of the Reactive Mani- festo in detail, and you will get to know several tools of the trade and the philosophy behind their design, enabling you to effectively use these tools to implement reactive designs. The design patterns that emerge from these tools are presented in the third part of the book. To set the stage for diving into the manifesto, we will first explore the challenges of creating a Reactive application, using the example of a well-known email service: we will imagine a reimplementation of Gmail. 1.1 The anatomy of a Reactive application The first task when starting such a project is to sketch an architecture for the deployment and draft the list of software artifacts that need to be developed. This may not be the final architecture, but you need to chart the problem space and explore potentially difficult aspects. We will start the Gmail example by enumerating the different high-level features of the application: The application must offer a view of the mailboxes to the user and display their contents. To this end, the system must store all emails and keep them available. It must allow the user to compose and send email. To make this more comfortable, the system should offer a list of contacts and allow the user to manage them. A good search function for locating emails is required. The real Gmail application has more features, but this list will suffice for our purposes. Some of these features are more intertwined than the others: for example, dis- playing emails and composing them are both part of the user interface and share (or compete for) the same screen space, whereas the implementation of email storage is only distantly related to these two. The implementation of the search function will need to be closer to the storage than the front-end presentation. Coping with load 5

Storage

Search Full search

Contacts Pop-up card Autocomplete Fuzzy index

Sign-on Editing

Gmail

Profile Listing

Mail Composing

Filters

Storage

Figure 1.2 Partially decomposed module hierarchy of the hypothetical Gmail implementation

These considerations guide the hierarchical decomposition of Gmail’s overall functionality into smaller and smaller pieces. More precisely, you can apply the Simple Com- ponent pattern as described in chapter 12, making sure you clearly delimit and segregate the different responsibilities of the entire application. The Error Kernel pattern and the Let-It-Crash pattern complement this process, ensuring that the application’s architecture is well suited to reliable failure handling—not only in case of machine or network outages, but also for rare failure conditions in the source code that are handled incorrectly (a.k.a. bugs). The result of this process will be a hierarchy of components that need to be developed and deployed. An example is shown in figure 1.2. Each component may be complex in terms of its function, such as the implementation of search algorithms; or it may be complex in its deployment and orchestration, such as providing email storage for billions of users. But it will always be simple to describe in terms of its responsibility. 1.2 Coping with load The resources necessary to store all those emails will be enormous: hundreds of millions of users with gigabytes of emails each will need exabytes3 of storage capacity. This magnitude of persistent storage will need to be provided by many distributed

3 One exabyte is 1 billion gigabytes (using decimal SI prefixes; using binary SI prefixes, one EB is roughly 1.07 billion GB). 6 CHAPTER 1 Why Reactive?

machines. No single storage device offers so much space, and it would be unwise to store everything in one location. Distribution makes the dataset resilient against local perils like natural disasters; but, more important, it also allows the data to be accessed efficiently from a larger region. For a worldwide user base, the data should be globally distributed as well. It would be preferable to have the emails of a Japanese user stored in or close to Japan (assuming that is where the user logs in from most of the time). This insight leads us to the Sharding pattern described in chapter 17: you can split up the overall dataset into many small pieces—or shards—that you then distribute. Because the number of shards is much smaller than the number of users, it is practical to make the location of each shard known throughout the system. In order to find a user’s mailbox, you only need to identify the shard it belongs to. You can do that by equipping every user with an ID that expresses geographical affinity (for example, using the first few digits to denote the country of residence), which is then mathematically partitioned into the correct number of shards (for example, shard 0 contains IDs 0–999,999; shard 1 contains IDs 1,000,000–1,999,999; and so on). The key here is that the dataset naturally consists of many independent pieces that can easily be separated from each other. Operations on one mailbox never affect another mailbox directly, so the shards also do not need to communicate among themselves. Each serves only one particular part of the solution. Another area in which the Gmail application will need a lot of resources is in the display of folders and emails to the user. It would be impossible to provide this functionality in a centralized fashion, not only for reasons of latency (even at the speed of light, it takes noticeable time to send information around the globe) but also due to the sheer number of interactions that millions of users perform every second. Here, you will also split the work among many machines, starting with the users’ computers: most of the graphical presentation is rendered within the browser, shifting the work- load very close to where it is needed and in effect sharding it for each user. The web browser will need to get the raw information from a server, ideally one that is close by to minimize network round-trip time. The task of connecting a user with their mailbox and routing requests and responses accordingly is one that can also easily be sharded. In this case, the browser’s network address directly provides all needed characteristics, including an approximate geographic location. One noteworthy aspect is that in all the aforementioned cases, resources can be added by making the shards smaller, distributing the load over more machines. The maximum number is given by the number of users or used network addresses, which will be more than enough to provide sufficient resources. This scheme will need adjustment only when serving a single user requires more computing power than a single machine can provide, at which point a user’s dataset or computing problem needs to be broken down into smaller pieces. This means that by splitting a system into distributable parts, you gain the ability to scale the service capacity, using a larger number of shards to serve more users. As long as the shards are independent from each other, the system is in theory infinitely Coping with failures 7

scalable. In practice, the orchestration and operation of a worldwide deployment with millions of nodes requires substantial effort and must of course be worth it. 1.3 Coping with failures Sharding datasets or computational resources solves the problem of providing sufficient resources for the nominal case, when everything is running smoothly and networks are operational. In order to cope with failures, you need the ability to keep running when things go wrong: A machine may fail temporarily (for example, due to overheating or kernel panic) or permanently (electrical or mechanical failure, fire, flood, and so on). Network components may fail, both within a computing center as well as outside on the internet—including the case that intercontinental overseas cables go down, resulting in a split of the internet into disconnected regions. Human operators or automated maintenance scripts may accidentally destroy parts of the data. The only solution to this problem is to replicate the system—its data or functionality—in more than one location. The geographical placement of the replicas needs to match the scope of the system; a global email service should serve each customer from multiple countries, for example. Replication is a more difficult and diverse topic than sharding because intuitively you mean to have the same data in multiple places—but keeping the replicas synchronized to match this expectation comes at a high cost. Should writing to the nearest location fail or be delayed if a more distant replica is momentarily unavailable? Should it be impossible to see the old data on a distant replica after the nearest one has already signaled completion of the operation? Or should such inconsistency just be unlikely or very short-lived? These questions will be answered differently between projects or even for different modules of one particular system. Therefore, you are presented with a spectrum of solutions that allows you to make trade-offs between operational complexity, performance, availability, and consistency. We will discuss several approaches covering a wide range of characteristics in chapter 13. The basic choices are as follows: Active–passive replication—Replicas agree on which one of them can accept updates. Fail-over to a different replica requires consensus among the remain- ing ones when the active replica no longer responds. Consensus-based multiple-master replication—Each update is agreed on by sufficiently many replicas to achieve consistent behavior across all of them, at the cost of availability and latency. Optimistic replication with conflict detection and resolution—Multiple active replicas disseminate updates and roll back transactions during conflict or discard conflicting updates that were performed during a network partition. 8 CHAPTER 1 Why Reactive?

Conflict-free replicated data types—This approach prescribes merge strategies such that conflicts cannot arise by definition, at the cost of providing only eventual consistency and requiring special care when creating the data model. In the Gmail example, several services should provide consistency to the user: if a user successfully moves an email to a different folder, they expect it to stay in that folder regardless of which client they use to access their mailboxes. The same goes for changes to a contact’s telephone number or the user's profile. For these data, you could use active–passive replication to keep things simple by making the failure response actions coarse-grained—that is, on a per-replica scope. Or you could use optimistic replication under the assumption that a single user will not concurrently make conflicting changes to the same data item—but keep in mind that this is a fair assumption only for human users. Consensus-based replication is needed within the system as an implementation detail of sharding by user ID, because the relocation of a shard must be recorded accu- rately and consistently for all clients. It would lead to user-visible distortions like an email disappearing and then reappearing if a client were to flip-flop between decom- missioned and live replicas. 1.4 Making the system responsive The previous two sections introduced reasons for distributing the system across several machines, computing centers, or possibly even continents, matching the scope and reliability requirements of the application. The foremost purpose of this exercise is to build an email service for end users, though, and for them the only metric that counts is whether the service does what they need when they need it. In other words, the application must respond quickly to any request a user makes. The easiest way to achieve this is, of course, to write an application that runs locally and that has all emails stored on the local machine as well: going across the network to fetch an answer will always take longer and be less reliable than having the answer close by. There is, thus, a tension between the need to distribute and the need to stay responsive. All distribution must be justified, as in the Gmail example. Where distribution is necessary, you encounter new challenges in the quest for responsiveness. The most annoying behavior of many distributed applications today is that their user interaction grinds to a halt when network connectivity is poor. Interest- ingly, it seems much simpler to deal with the complete absence of a connection than with a trickling flow of data. One pattern that is helpful in this context is the Circuit Breaker pattern discussed in detail in chapter 12. With this tool, you can monitor the availability and performance of a service that you are calling on for some function so that when the quality falls below a threshold (either too many failures or too long a response latency), the circuit breaker trips, forcing a switch to a mode where that service is not used. The unavailability of parts of the system needs to be considered from the beginning; the Circuit Breaker pattern addresses this concern. Making the system responsive 9

Another threat to responsiveness arises when a service that the application depends on becomes momentarily overloaded. A backlog of requests will accumulate, and while these are processed, response latencies will be much longer than normal. This situation can be avoided by employing flow control, as described in chapter 16. In the Gmail example, there are several points at which circuit breakers and flow control are needed: Between the front end that runs on the users’ devices and the web servers that provide access to back-end functionality Between the web servers and back-end services The reason for the first point has already been mentioned: the desire to keep the user- visible part of the application responsive under all conditions, even if sometimes the only thing it can do is signal that the server is down and that the request will be completed at a later time. Depending on how much functionality can or should practically be duplicated in the front end for this offline mode, some areas of the user interface may need to be deactivated. The reason for the second point is that the front end would otherwise need to have different circuit breakers for different kinds of requests to the web server, each circuit breaker corresponding to the specific subset of back-end services needed by one kind of request. Switching the entire application to offline mode when only a small part of the back-end services are unavailable would be an unhelpful over- response. Tracking this in the front end would couple its implementation to the precise structure of the back end, requiring the front-end code to be changed whenever the service composition of the back end was altered. The web-server layer should hide these details and provide its clients with responses as quickly as possible under all circumstances. Take, for example, the back-end service that provides the information shown on the contact card that pops up when hovering the pointer over an email sender’s name. This is a nonessential function, considering the overall function of Gmail, so the web server may return a temporary failure code for such requests while that back- end service is unavailable. The front end does not need to track this state; it can merely refrain from showing the pop-up card and retry the request when interaction with the user triggers it again. This reasoning applies not only at the web server layer. In a large application that consists of hundreds or thousands of back-end services, it is imperative to confine the treatment of failure and unavailability in this fashion; otherwise, the system would be unreasonable in the sense that its behavior could no longer be understood by humans. Just as functionality is modularized, the treatment of failure conditions must be encapsulated in comprehensible scopes as well. 10 CHAPTER 1 Why Reactive?

1.5 Avoiding the ball of mud The Gmail application at this point consists of a front-end part that runs on the user’s device, back-end services that provide storage and functionality, and web servers that act as entry points into the back end. The latter serve an important purpose beyond the responsiveness discussed in the previous section: they decouple the front end from the back end architecturally. Having this clearly defined ingress point for client requests makes it simpler to reason about the interplay between the part of the application that runs on the users’ devices and the part that runs on servers in the cloud. The back end so far consists of a multitude of services whose partitioning and relationships resulted from the application of the Simple Component pattern. By itself, this pattern does not provide the checks and balances that keep the architecture from devolving into a large mess where every service talks with almost every other service. Such a system would be hard to manage even with perfect individual failure handling, circuit breakers, and flow control; it certainly would not be possible for a human to understand it in its entirety and confidently make changes to it. This scenario has informally been called the big ball of mud. With the problem lying in the unrestrained interaction between arbitrary back-end services, the solution is to focus on the communication paths within the entire application and to specifically design them. This is called message flow and is discussed in detail in chapter 15. The service decomposition shown in figure 1.2 is too coarse-grained to serve as an example for a “ball of mud,” but an illustration for the principle of message-flow design would be that the service that handles email composition probably should not talk directly to the contact pop-up service: if composing an email entails showing the contact card of someone mentioned in the email, then instead of making the back end responsible for that, the front end should ask for the pop-up, just as it does when the user hovers the mouse pointer over an email header. In this way, the number of possible message-flow paths is reduced by one, making the overall interaction model of back-end services a little simpler. Another benefit of carefully considering message flow lies in facilitating testing and making it easier to ensure coverage of all interaction scenarios. With a comprehensive message-flow design, it is obvious which other services a component interacts with and what is expected from the component in terms of throughput and latency. This can be turned around and used as a canary in the coal mine: whenever it is difficult to assess which scenarios should be tested for a given component, that is a sign that the system is in danger of becoming a big ball of mud. 1.6 Integrating nonreactive components The final important aspect of creating an application according to Reactive principles is that it will, in most cases, be necessary to integrate with existing systems or infrastructure that does not provide the needed characteristics. Examples are device driv- ers that lack encapsulation (for example, by terminating the entire process in case of Summary 11

failure), APIs that execute their effects synchronously and thereby block the caller from reacting to other inputs or timeouts in the meantime, systems with unbounded input queues that do not respect bounded response latency, and so on. Most of these issues are dealt with using the resource-management patterns discussed in chapter 14. The basic principle is to retrofit the needed encapsulation and asynchronous boundaries by interacting with the resource within a dedicated Reactive component, using extra threads, processes, or machines as necessary. This allows these resources to be integrated seamlessly into the architecture. When interfacing with a system that does not provide bounded response latency, it is necessary to retrofit the ability to signal momentary overload situations. This can to some degree be achieved by employing circuit breakers, but in addition you must consider what the response to overload should be. The flow-control patterns described in chapter 16 help in this case as well. An example in the context of the Gmail application is a hypothetical integration with an external utility, such as a shared shopping list. Within the Gmail front end, the user can add items to the shopping list by extracting the needed information semiautomatically from emails. This function would be supported in the back end by a service that encapsulates the external utility’s API. Assuming that the interaction with the shopping list requires the use of a native library that is prone to crash and bring down the process it is running in, it is desirable to dedicate a process to this task alone. This encapsulated form of the external API is then integrated via the operating system’s interprocess communication (IPC) facilities, such as pipes, sockets, and shared memory. Assuming further that the shopping list’s implementation employs a practically unbounded input queue, you need to consider what should happen when latencies increase. For example, if it takes minutes for an item to show up on the shopping list, users will be confused and perhaps frustrated. A solution to this problem would be to monitor the shopping list and observe the latency from the Gmail back-end service that manages this interaction. When the currently measured latency exceeds the acceptable threshold, the service will either respond to requests with a rejection and a temporary failure code, or perform the operation and include a warning notice in the response. The front-end application can then notify the user of either outcome: in one case it suggests retrying later, and in the other it informs them about the delay. 1.7 Summary In this chapter, we explored the Reactive landscape in the context of the principles laid out in the Reactive Manifesto and surveyed the main challenges facing you when building applications in this style. For a more detailed example of designing a Reac- tive application, please refer to appendix B. The next chapter takes a deep dive into the manifesto itself, providing a detailed discussion of the points that are condensed into a compressed form in appendix C. A walk-through of the Reactive Manifesto

This chapter introduces the manifesto in detail: where the original text is as short as possible and rather dense, we unfold it and discuss it in great depth. For more background information on the theory behind the manifesto, please refer to part 2 of the book. 2.1 Reacting to users So far, this book has used the word user informally and mostly in the sense of humans who interact with a computer. You interact only with your web browser in order to read and write emails, but many computers are needed in the background to perform these tasks. Each of these computers offers a certain set of services, and the consumer or user of these services will in most cases be another computer that is acting on behalf of a human, either directly or indirectly. The first layer of services is provided by the front-end server and consumed by the web browser. The browser makes requests and expects responses—predomi- nantly using HTTP, but also via WebSockets. The resources that are requested can pertain to emails, contacts, chats, searching, and many more (plus the definition of the styles and layout of the website). One such request might be related to the images of people you correspond with: when you hover over an email address, a pop-up window appears that contains details about that person, including a photograph or an avatar image. In order to render that image, the web browser makes a request to the front-end server. Figure 2.1 shows how this might be implemented using a traditional servlets approach.

12 Reacting to users 13

System

Application

Controller Cache

Connection Image Fallback pool database

Figure 2.1 The front-end server for images first checks an in-memory cache, then attempts to retrieve the image from storage, and finally returns a fallback image if neither is successful.

The user action of hovering the mouse pointer over an email address sets in motion a flurry of requests via the web browser, the front-end server, and the internal image service down to the storage system, followed by their respective responses traveling in the opposite direction until the image is properly rendered on the screen. Along this chain are multiple relationships from user to service, and all of them need to meet the basic challenges outlined in the introduction; most important is the requirement to respond quickly to each request. When designing the overall implementation of a feature like the image service, you need to think about services and their users’ requirements not only on the outside but also on the inside. This is the first part of what it means to build reactive applications. Once the system has been decomposed in this way, you need to turn your focus to making these services as responsive as necessary to satisfy their users at all levels. To understand why Reactive systems are better than the traditional alternatives, it is useful to examine a traditional implementation of an image service. Even though it has a cache, a connection pool, and even a fallback image for when things go wrong, it can fail badly when the system is stressed. Understanding how and why it fails requires looking beyond the single-thread illusion. Once you understand the failures, you will see that even within the confines of a traditional framework, you can improve the image service with a simplified version of the Managed Queue pattern that is covered in chapter 16.

2.1.1 Understanding the traditional approach We will start with a naive implementation to retrieve an image from a database. The application has a controller that first checks a cache to see whether the image has 14 CHAPTER 2 A walk-through of the Reactive Manifesto

been retrieved recently. If the controller finds the image in the cache, it returns the image right away. If not, it tries to retrieve the image from the database. If it finds the image there, the image is added to the cache and also returned to the original requester. If the image is not found, a static fallback image is returned, to avoid presenting the user with an error. This pattern should be familiar to you. This simplistic controller might contain code like the following.

Listing 2.1 Excerpt from a simple controller for an image service

public interface Images { Image get(String Key); void add(String key, Image image); } public Images cache; Assumed thread-safe public Images database; Wraps a database Image result = cache.get(key); connection pool if (result != null) { return result; Image is found in the cache }else{ result = database.get(key); Image is found in the if (result != null) { database, added to the cache, cache.add(key, result); and returned to the client return result; }else{ return fallback; Image is not retrieved } from the database }

At the next level of detail, the application may be built on a framework that has some ability to handle concurrency, such as Java servlets. When a new request is received, the application framework assigns it to a request thread. That thread is then responsible for carrying the request through to a response. The more request threads are con- figured, the more simultaneous requests the system is expected to handle. On a cache hit, the request thread can provide a response immediately. On a cache miss, the internal implementation of Images needs to obtain a connection from the pool. The database query itself may be performed on the request thread, or the connection pool may use a separate thread pool. Either way, the request thread is obliged to wait for the database query to complete or time out before it can fulfill the request. When you are tuning the performance of a system such as this, one of the key parameters is the ratio of request threads to connection-pool entries. There is not much point in making the connection pool larger than the request-thread pool. If it is the same size and all the request threads are waiting on database queries, the system may find itself temporarily with little to do other than wait for the database to respond. That would be unfortunate if the next several requests could have been served from the cache; instead of being handled immediately, they will have to wait for Reacting to users 15

an unrelated database query to complete so that a request thread will become available. On the other hand, setting the connection pool too small will make it a bottle- neck; this risks the system being limited by request threads stuck waiting for a connection. The best answer for a given load is somewhere between the extremes. The next section looks at finding a balance.

2.1.2 Analyzing latency with a shared resource The simplistic implementation can be analyzed first by examining one extreme consisting of an infinite number of request threads sharing a fixed number of database connections. Assume each database query takes a consistent time W to complete, and for now ignore the cache. We will revisit the effect of the cache in section 2.3.1, when we introduce Amdahl’s Law. You want to know how many database connections L will be used for a given load, which is represented as λ. A formula called Little’s Law gives the answer:

L = λ × W

Little’s Law is valid for the long-term averages of the three quantities independent of the actual timing with which requests arrive or the order in which they are processed. If the database takes on average 30 ms to respond, and the system is receiving 500 requests per second, you can apply Little’s Law:

L = 500 requests/second× 0.03 seconds/request L = The average number of connections being used will be 15, so you will need at least that many connections to keep up with the load. If there are requests waiting to be serviced, they must have some place to wait. Typ- ically, they wait in a queue data structure somewhere. As each request is completed, the next request is taken from the queue for processing. Referring to figure 2.2, you may notice that there is no explicit queue. If this were coded using traditional synchronous Java servlets, the queue would consist of an internal collection of request threads waiting for their turn with the database connection. On average, there would be 15 such threads waiting. That is not good, because, whereas a queue is a light- weight data structure, the request threads in the queue are relatively expensive resources. Worse, 15 is just the average: the peaks are much higher. In reality, the thread pool will not be infinite. If there are too many requests, they will spill back into the TCP buffer and eventually back to the browser, resulting in unhelpful errors rather than the desired fallback image. The first thing you might do is increase the number of entries in the database connection pool. As long as the database can continue to handle the resulting load, this will help the average case. The important thing to note is that you are still working with 16 CHAPTER 2 A walk-through of the Reactive Manifesto

System

Application

Controller Cache TCP buffer

Connection Image Fallback pool database

Figure 2.2 Using standard listener threads and a connection pool results in the listeners acting as queue entries, with overflow into the system TCP buffers.

average times. Real-world events can lead to failure modes that are far worse. For example, if the database stops responding at all for several minutes, 500 requests per second will overwhelm an otherwise sufficient thread pool. You need to protect the system.

2.1.3 Limiting maximum latency with a queue The initial implementation blocked and waited for a database connection to become available; it returned null only if the requested image was not found in the database. A simple change will add some protection: if a database connection is not available, return null right away. This will free the request thread to return the fallback image rather than stalling and consuming a large amount of resources. This approach couples two separate decisions into one: the number of database queries that can be accepted simultaneously is equal to the size of the connection pool. That may not be the result you want: it means the system will either return right away if no connection is available or return in 30 ms if one is available. Suppose you are willing to wait a bit longer in exchange for a much better rate of success. At this point, you can introduce an explicit queue, as shown in figure 2.3. Now, instead of returning right away if no connection is available, new requests are added to the queue. They are turned away only if the queue itself is full. The addition provides much better control over system behavior. For example, a queue with a maximum length of only 3 entries will respond in no more than a total of 120 ms, including 90 ms progressing through the queue and another 30 ms for the database query. The size of the queue provides an upper bound that you can control. Depending on the rate of requests, the average response may be lower, perhaps less Reacting to users 17

System

Application

Controller Cache

Explicit queue

Connection Image Fallback pool database

Figure 2.3 Adding an explicit queue to manage access to the database connection pool allows you to manage the maximum system latency separately from the listener thread pool size and the connection pool size. than 100 ms. If the cache that was ignored in the analysis is now considered, the average drops still further. With a 50% cache-hit rate, the image server could offer an average response time of less than 50 ms. Given what you know about how that 50 ms average is achieved, you also would know not to set a timeout less than 120 ms. If that time was not acceptable, the simpler solution would be to use a smaller queue. A developer who knows only that the average is less than 50 ms might assume it is a Gaussian distribution and be tempted to set a timeout value at perhaps 80 or 100 ms. Indeed, the assumptions that went into this analysis are vulnerable to the same error, because the assumption that the database provides a consistent 30 ms response time would be questionable in a real-world implementation. Real databases have caches of their own. Setting a timeout has the effect of choosing a boundary at which the system will be considered to have failed. Either the system succeeded or it failed. When viewed from that perspective, the average response time is less important than the maximum response time. Because systems typically respond more slowly when under heavy load, a timeout based on the average will result in a higher percentage of failures under load and will also waste resources when they are needed most. Choosing timeouts will be revisited in section 2.4 and again in chapter 11. For now, the important realization is that the average response time often has little bearing on choosing the maximum limits.

18 CHAPTER 2 A walk-through of the Reactive Manifesto

2.2 Exploiting parallelism The simplest case of a user–service relationship is invoking a method or function:

val result = f(42)

The user provides the argument “42” and hands over control of the CPU to the function f, which might calculate the 42nd Fibonacci number or the factorial of 42. What- ever the function does, you expect it to return some result value when it is finished. This means that invoking the function is the same as making a request, and the function returning a value is analogous to it replying with a response. What makes this example so simple is that most programming languages include syntax like this, which allows direct usage of the response under the assumption that the function does indeed reply. If that were not to happen, the rest of the program would not be executed, because it could not continue without the response. The underlying execution model is that the evaluation of the function occurs synchronously, on the same thread, and this ties the caller and the callee together so tightly that failures affect both in the same way. Sequential execution of functions is well supported by all popular programming languages out of the box, as illustrated in figure 2.4 and shown in this example using Java syntax:

ReplyAa=computeA(); ReplyBb=computeB(); ReplyCc=computeC(); Resultr=aggregate(a,b,c); The sequential model is easy to understand. It was adequate for early computers that had only one processing core, but it necessitates waiting for all the results to be computed by the same resource while other resources remain idle.

2.2.1 Reducing latency via parallelization In many cases, there is one possibility for latency reduction that immediately presents itself. If, for the completion of a request, several other services must be involved, then the overall result will be obtained more quickly if the other services can perform their

A B C

Task

Figure 2.4 A task consisting of three subtasks that are executed sequentially: the total response latency is given by the sum of the three individual latencies. Exploiting parallelism 19 functions in parallel, as shown in figure 2.5. This requires that no dependency exists such that, for example, task B needs the output of task A as one of its inputs, which frequently is the case. Take as an example the Gmail app in its entirety, which is composed of many different but independent parts. Or the contact information pop-up window for a given email address may contain textual information about that person as well as their image, and these can clearly be obtained in parallel. When performing subtasksA, B, and C sequentially, as shown in figure 2.4, the overall latency depends on the sum of the three individual latencies. With parallel execution, overall latency equals the latency of whichever of the subtasks takes lon- gest. In the implementation of a real social network, the number of subtasks can easily exceed 100, rendering sequential execution entirely impractical. Parallel execution usually requires some extra thought and library support. For one thing, the service being called must not return the response directly from the method call that initiated the request, because in that case the caller would be unable to do anything while task A was running, including sending a request to perform task B in the meantime. The way to get around this restriction is to return a Future of the result instead of the value itself:

Future a = taskA(); Future b = taskB(); Future c = taskC(); Resultr=aggregate(a.get(),b.get(),c.get()); A Future is a placeholder for a value that may eventually become available; as soon as it does, the value can be accessed via the Future object. If the methods invoking subtasks A, B, and C are changed in this fashion, then the overall task just needs to call them to get back one Future each. Futures are discussed in greater detail in the next chapter.

Task

Figure 2.5 A task consisting of three subtasks that are executed in parallel: the total response latency is given by the maximum of the three individual latencies. 20 CHAPTER 2 A walk-through of the Reactive Manifesto

The previous code snippet uses a type called Future that is defined in the Java standard library (in the package java.util.concurrent). The only method it defines for accessing the value is the blocking get() method. Blocking here means the calling thread is suspended and cannot do anything else until the value has become available. We can picture the use of this kind of Future like so (written from the perspective of the thread handling the overall task): When my boss gives me the task to assemble the overview file of a certain client, I will dispatch three runners: one to the client archives to fetch the address, photograph, and contract status; one to the library to fetch all articles the client has written; and one to the post office to collect all new messages for this client. This is a vast improvement over having to perform these tasks myself, but now I need to wait idly at my desk until the runners return, so that I can collate everything they bring into an envelope and hand that back to my boss. It would be much nicer if I could leave a note telling the runners to place their findings in the envelope and telling the last one to come back to dispatch another runner to hand it to my boss without involving me. That way I could handle many more requests and would not feel useless most of the time.

2.2.2 Improving parallelism with composable Futures What the developer should do is describe how the values should be composed to form the final result and let the system find the most efficient way to compute the values. This is possible with composable Futures, which are part of many programming languages and libraries, including newer versions of Java (CompletableFuture is introduced in JDK 8). Using this approach, the architecture turns completely from synchronous and blocking to asynchronous and nonblocking; the underlying machin- ery needs to become task-oriented in order to support this. The result is far more expressive than the relatively primitive precursor, the callback. The previous example transforms into the following, using Scala syntax:1

val fa: Future[ReplyA] = taskA() val fb: Future[ReplyB] = taskB() val fc: Future[ReplyC] = taskC() val fr: Future[Result] = for (a <- fa; b <- fb; c <- fc) yield aggregate(a, b, c) Initiating a subtask as well as its completion are just events that are raised by one part of the program and reacted to in another part: for example, by registering an action to be taken with the value supplied by a completed Future. In this fashion, the latency of the method call for the overall task does not even include the latencies for subtasks A, B, and C, as shown in figure 2.6. The system is free to handle other requests while

1 This would also be possible with the Java 8 CompletionStage using the andThen combinator, but due to the lack of for-comprehensions, the code would grow in size relative to the synchronous version. The Scala expression on the last line transforms to corresponding calls to flatMap, which are equivalent to CompletionStage’s andThen. Exploiting parallelism 21

Task B Result

Figure 2.6 A task consisting of three subtasks that are executed as Futures: the total response latency is given by the maximum of the three individual latencies, and the initiating thread does not need to wait for the responses.

those are being processed, eventually reacting to their completion and sending the overall response back to the original user. An added benefit is that additional events like task timeouts can be added without much hassle, because the entire infrastructure is already there. It is entirely reasonable to perform task A, couple the resulting Future with one that holds a TimeoutException after 100 ms, and use the combined result in the processing that follows. Then, either of the two events—completion of A or the timeout—triggers the actions that were attached to the completion of the combined Future.

THE NEED FOR ASYNCHRONOUS RESULT COMPOSITION You may be wondering why this second part—asynchronous result composition—is necessary. Would it not be enough to reduce response latency by exploiting parallel execution? The context of this discussion is achieving bounded latency in a system of nested user–service relationships, where each layer is a user of the service beneath it. Because parallel execution of the subtasks A, B, and C depends on their initiating methods returning Futures instead of strict results, this must also apply to the overall task. That task is very likely part of a service that is consumed by a user at a higher level, and the same reasoning applies on that higher level as well. For this reason, it is imperative that parallel execution be paired with asynchronous and task-oriented result aggregation.

Composable Futures cannot be fully integrated into the image server example discussed earlier using the traditional servlet model. The reason is that the request thread encapsulates all the details necessary to return a response to the browser. There is no mechanism to make that information available to a future result. This is addressed in Servlet 3 with the introduction of AsyncContext.

2.2.3 Paying for the serial illusion Traditionally, ways of modeling interactions between components—like sending to and receiving from the network—are expressed as blocking API calls: 22 CHAPTER 2 A walk-through of the Reactive Manifesto

final Socket socket = ... socket.getOutputStream.write(requestMessageBytes); final int bytesRead = socket.getInputStream().read(responseBuffer);

Each of these blocking calls interacts with the network equipment, generating messages and reacting to messages under the hood, but this fact is completely hidden in order to construct a synchronous façade on top of the underlying message-driven system. The thread executing these commands will suspend its execution if not enough space is available in the output buffer (for the first line) or if the response is not immediately available (on the second line). Consequently, this thread cannot do any other work in the meantime: every activity of this type that is ongoing in parallel needs its own thread, even if many of those are doing nothing but waiting for events to occur. If the number of threads is not much larger than the number of CPU cores in the system, then this does not pose a problem. But given that these threads are mostly idle, you want to run many more of them. Assuming that it takes a few microseconds to prepare the requestMessageBytes and a few more microseconds to process the responseBuffer, whereas the time for traversing the network and processing the request on the other end is measured in milliseconds, it is clear that each thread spends more than 99% of its time in a waiting state. In order to fully utilize the processing power of the available CPUs, this means running hundreds if not thousands of threads, even on commodity hardware. At this point, you should note that threads are managed by the operating system kernel for efficiency reasons.2 Because the kernel can decide to switch out threads on a CPU core at any point in time (for example, when a hardware interrupt happens or the time slice for the current thread is used up), a lot of CPU state must be saved and later restored so that the running application does not notice that something else was using the CPU in the meantime. This is called a context switch and costs thousands of cycles3 every time it occurs. The other drawback of using large numbers of threads is that the scheduler—the part of the kernel that decides which thread to run on which CPU core at any given time—will have a hard time finding out which threads are run- nable and which are waiting and then selecting one such that each thread gets its fair share of the CPU. The takeaway of the previous paragraph is that using synchronous, blocking APIs that hide the underlying message-driven structure wastes CPU resources. If messages were made explicit in the API such that instead of suspending a thread, you would just suspend the computation—freeing up the thread to do something else—then this overhead would be reduced substantially. The following example shows (remote) messaging between Akka Actors from Java 8:

2 Multiplexing several logical user-level threads on a single OS thread is called a many-to-one model or green threads. Early JVM implementations used this model, but it was abandoned quickly (http://docs.oracle .com/cd/E19455-01/806-3461/6jck06gqh/index.html). 3 Although CPUs have gotten faster, their larger internal state has negated the advances made in pure execution speed such that a context switch has taken roughly 1 µs without much improvement for two decades. The limits of parallel execution 23 Sends a message to the actor CompletionStage future = reference, using a CompletionStage ask(actorRef, request, timeout) as the destination for the response .thenApply(Response.class::cast); Maps the response to its expected future.thenAccept(response -> ); type, failing upon mismatch Registers further processing to be done once a response is received and mapped

Here, the sending of a request returns a handle to the possible future reply—a composable Future, as discussed in chapter 3—to which a callback is attached that runs when the response has been received. Both actions complete immediately, letting the thread do other things after having initiated the exchange. 2.3 The limits of parallel execution Loose coupling between components—by design as well as at runtime—includes another benefit: more efficient execution. Although hardware used to increase capacity primarily by increasing the computing power of a single sequential execution core, physical limits4 began impeding progress on this front around 2006. Modern processors now expand capacity by adding ever more cores, instead. In order to benefit from this kind of growth, you must distribute computation even within a single machine. When using a traditional approach with shared state concurrency based on mutual exclusion by way of locks, the cost of coordination between CPU cores becomes very significant.

2.3.1 Amdahl’s Law The example in section 2.1 includes an image cache. The most likely implementation would be a map shared among the request threads running on multiple cores in the same JVM. Coordinating access to a shared resource means executing those portions of the code that depend on the integrity of the map in some synchronized fashion. The map will not work properly if it is being changed at the same time it is being read. Operations on the map need to happen in a serialized fashion in some order that is globally agreed on by all parts of the application; this is also called sequential consistency. There is an obvious drawback to such an approach: portions that require synchronization cannot be executed in parallel. They run effectively single-threaded. Even if they execute on different threads, only one can be active at any given point in time. The effect this has on the possible reduction in runtime that is achievable by parallelization is captured by Amdahl’s Law, shown in figure 2.7.

T() N Figure 2.7 Amdahl’s Law specifies the Sn()==------=------TN() – α + α ()N – maximum increase in speed that can be achieved α + ------by adding additional threads. N

4 The finite speed of light as well as power dissipation make further increases in clock frequency impractical. 24 CHAPTER 2 A walk-through of the Reactive Manifesto

95% parallel 20

12 90% parallel 10 Speedup 8

6 75% parallel 4 50% parallel 2

0 02 16 128 1024 8192 Processor

Figure 2.8 The increase in speed of a program using multiple processors in parallel computing is limited by the sequential fraction of the program. For example, if 95% of the program can be parallelized, the theoretical maximum speedup using parallel computing will be 20 times, no matter how many processors are used.

Here, N is the number of available threads, α is the fraction of the program that is serialized, and T(N)is the time the algorithm needs when executed with N threads. This formula is plotted in figure 2.8 for different values of α across a range of available threads—they translate into the number of CPU cores on a real system. You will notice that even if only 5% of the program runs inside these synchronized sections, and the other 95% is parallelizable, the maximum achievable gain in execution time is a factor of 20; getting close to that theoretical limit would mean employing the ridiculous number of about 1,000 CPU cores.

2.3.2 Universal Scalability Law Amdahl’s Law also does not take into account the overhead incurred for coordinating and synchronizing the different execution threads. A more realistic formula is provided by the Universal Scalability Law,5 shown in figure 2.9.

Figure 2.9 The Universal Scalability Law provides the N Sn()= ------maximum increase in speed that can be achieved by ++α()N – βNN()– adding additional threads, with an additional factor to account for coordination.

5 N. J. Gunther, “A Simple Capacity Model of Massively Parallel Transaction Systems,” 2003, www.perfdynamics .com/Papers/njgCMG93.pdf. See also “Neil J. Gunther: Universal Law of Computational Scalability,” Wikipe- dia, https://en.wikipedia.org/wiki/Neil_J._Gunther#Universal_Law_of_Computational_Scalability. The limits of parallel execution 25

18 no coherency cost 0.2% coherency cost 16 0.5% coherency cost 14 1% coherency cost 5% coherency cost 12

Performance increase 6

0 1 10 100 1000 10000 CPU cores

Figure 2.10 At some point, the increase in speed from adding more resources is eaten up by the cost of maintaining coherency within the system. The precise point depends on the parallel program fraction and the time spent on coherency.

The Universal Scalability Law adds another parameter describing the fraction of time spent ensuring that the data throughout the system is consistent. This factor is called the coherency of the system, and it combines all the delays associated with coordinating between threads to ensure consistent access to shared data structures. This new term dominates the picture when you have a large number of cores, taking away the throughput benefits and making it unattractive to add more resources beyond a certain point. This is illustrated in figure 2.10 for rather low assumptions on the coherency parameter; distributed systems will spend considerably more than a small percentage of their time on coordination. The conclusion is that synchronization fundamentally limits the scalability of your application. The more you can do without synchronization, the better you can distribute your computation across CPU cores—or even network nodes. The optimum would be to share nothing—meaning no synchronization would be necessary—in which case scalability would be perfect. In figure 2.9, α and β would be zero, simplifying the entire equation to

Sn()= n

In plain words, this means that using n times as many computing resources, you achieve n times the performance. If you build your system on fully isolated compartments that are executed independently, then this will be the only theoretical limit, assuming you can split the task into at least n compartments. In practice, you need to exchange requests and responses, which requires some form of synchronization as well, but the cost of that is very low. On commodity hardware, it is possible to exchange several hundred million messages per second between CPU cores. 26 CHAPTER 2 A walk-through of the Reactive Manifesto

2.4 Reacting to failure The previous sections concern designing a service implementation such that every request is met with a response within a given time. This is important because otherwise the user cannot determine whether the request has been received and processed. But even with flawless execution of this design, unexpected things will happen eventually: Software will fail. There will always be that exception you forgot to handle (or that was not documented by the library you are using); or you may get synchronization only a tiny bit wrong, causing a deadlock to occur; or the condition you formulated for breaking a loop may not cope with a weird edge case. You can always trust the users of your code to figure out ways to eventually find all these failure conditions and more. Hardware will fail. Everyone who has operated computing hardware knows that power supplies are notoriously unreliable; that hard disks tend to turn into expensive door stops, either during the initial burn-in phase or after a few years; and that dying fans lead to the silent death of all kinds of components by overheating them. In any case, your invaluable production server will, according to Murphy’s Law, fail exactly when you most need it. Humans will fail. When you task maintenance personnel with replacing a failed hard disk in RAID5, a study 6 finds that there is a 10% chance that they will replace the wrong one, leading to the loss of all data. An anecdote from Roland’s days as a network administrator is that cleaning personnel unplugged the power of the main server for the workgroup—both redundant cords at the same time—in order to connect the vacuum cleaner. None of these things should happen, but it is human nature that you will have a bad day from time to time. Timeout is failure. The reason for a timeout may not be related to the internal behavior of the system. For example, network congestion can delay messages between components of your system even when all the components are functioning normally. The source of delay may be some other system that shares the network. From the perspective of handling an individual request, it does not matter whether the cause is permanent or transient. The fact is that the one request has taken too long and therefore has failed. The question therefore is not if a failure occurs but only when or how often. The user of a service does not care how an internal failure happened or what exactly went wrong, because the only response the user will get is that no normal response is received. Connections may time out or be rejected, or the response may consist of an opaque internal error code. In any case, the user will have to carry on without the response, which for humans probably means using a different service: if you try to book a flight

6 Aaron B. Brown (IBM Research), “Oops! Coping with Human Error,” ACM Queue 2, no. 8 (Dec. 6, 2004), http://queue.acm.org/detail.cfm?id=1036497. Reacting to failure 27 and the booking site stops responding, then you will take your business elsewhere and probably not come back anytime soon (or, in a different business, like online banking, users will overwhelm the support hotline). A high-quality service is one that performs its function very reliably, preferably without any downtime at all. Because failure of computer systems is not an abstract possibility but is in fact certain, the question arises: how can you hope to construct a reliable service? The Reactive Manifesto chooses the term resilience instead of reliability precisely to capture this apparent contradiction.

What does resilience mean? Merriam-Webster defines resilience as follows: The ability of a substance or object to spring back into shape The capacity to recover quickly from difficulties

The key notion here is to aim at fault tolerance instead of fault avoidance, because avoidance will not be fully successful. It is of course good to plan for as many failure scenarios as you can, to tailor programmatic responses such that normal operations can be resumed as quickly as possible—ideally without the user noticing anything. The same must also apply to those failure cases that were not foreseen and explicitly accommodated in the design, knowing that these will happen as well. But resilience goes one step further than fault tolerance: a resilient system not only withstands a failure but also recovers its original shape and feature set. As an example, consider a satellite that is placed in orbit. In order to reduce the risk of losing the mis- sion, every critical function is implemented at least twice, be it hardware or software. For the case that one component fails, there are procedures that switch to the backup component. Exercising such a fail-over keeps the satellite functioning, but from then on the affected component will not tolerate additional faults because there was only one backup. This means the satellite subsystems are fault tolerant but not resilient. There is only one generic way to protect your system from failing as a whole when a part fails: distribute and compartmentalize. The former can informally be translated as “don’t put all your eggs in one basket,” and the latter adds “protect your baskets from one another.” When it comes to handling a failure, it is important to delegate, so that the failed compartment itself is not responsible for its own recovery. Distribution can take several forms. The one you probably think of first involves replicating an important database across several servers such that, in the event of a hardware failure, the data are safe because copies are readily available. If you are really concerned about those data, then you may go as far as placing the replicas in different buildings in order not to lose all of them in the case of fire—or to keep them independently operable when one of them suffers a complete power outage. For the really paranoid, those buildings would need to be supplied by different power grids, better yet in different countries or on separate continents. 28 CHAPTER 2 A walk-through of the Reactive Manifesto

2.4.1 Compartmentalization and bulkheading The further apart the replicas are kept, the smaller the probability of a single fault affecting all of them. This applies to all kinds of failures, whether software, hardware, or human: reusing one computing resource, operations team, set of operational procedures, and so on creates a coupling by which multiple replicas can be affected synchronously or similarly. The idea behind this is to isolate the distributed parts or, to use a metaphor from ship building, to use bulkheading. Figure 2.11 shows the schematic design of a large cargo ship whose hold is separated by bulkheads into many compartments. When the hull is breached for some reason, only those compartments that are directly affected will fill up with water; the others will remain properly sealed, keeping the ship afloat.

Figure 2.11 The term bulkheading comes from ship building and means the vessel is segmented into fully isolated compartments.

One of the first examples of this building principle was the Titanic, which featured 15 bulkheads between bow and stern and was therefore considered unsinkable.7 That particular ship did in fact sink, so what went wrong? In order to not inconvenience passengers (in particular the higher classes) and to save money, the bulkheads extended only a few feet above the water line, and the compartments were not sealable at the top. When five compartments near the bow were breached during the collision with the ice- berg, the bow dipped deeper into the water, allowing the water to flow over the top of the bulkheads into more and more compartments until the ship sank. This example—although certainly one of the most terrible incidents in marine history—perfectly demonstrates that bulkheading can be done wrong in such a way that it becomes useless. If the compartments are not truly isolated from each other, failure can cascade among them to bring down the entire system. One example from distributed computing designs is managing fault tolerance at the level of entire application servers, where one failure can lead to the failure of other servers by overloading or stalling them. Modern ships employ full compartmentalization where the bulkheads extend from keel to deck and can be sealed on all sides, including the top. This does not make the ships unsinkable, but in order to obtain a catastrophic outcome, the ship needs to be mismanaged severely and run with full speed against a rock.8 That metaphor translates in full to computer systems.

7 “There is no danger that Titanic will sink. The boat is unsinkable and nothing but inconvenience will be suf- fered by the passengers.” —Phillip Franklin, White Star Line vice president, 1912. 8 See, for example, the Costa Concordia disaster: https://en.wikipedia.org/wiki/Costa_Concordia_disaster. Reacting to failure 29

2.4.2 Using circuit breakers No amount of planning and optimization will guarantee that the services you implement or depend on abide by their latency bounds. We will talk more about the nature of the things that can go wrong when discussing resilience, but even without knowing the source of the failure, there are some useful techniques for dealing with services that violate their bounds. When users are momentarily overwhelming a service, then its response latency will rise, and eventually it will start failing. Users will receive their responses with more delay, which in turn will increase their own latency until they get close to their own limits. In the image server example in section 2.1.2, you saw how adding an explicit queue protected the client by rejecting requests that would take more than the acceptable response time to service. This is useful when there is a short spike in demand for the service. If the image database were to fail completely for several minutes, the behavior would not be ideal. The queue would fill with a backlog of requests that, after a short time, would be useless to process. A first step would be to cull the old queue entries, but the queue would refill immediately with still more queries that would take too long to process. In order to stop this effect from propagating across the entire chain of user–service relationships, users need to shield themselves from the overwhelmed service during such time periods. The way to do this is well known in electrical engineering: install a circuit breaker, as shown in figure 2.12. The idea here is simple: when involving another service, monitor the time it takes for the response to come back. If the time is consistently greater than the allowed threshold this user has factored into its own latency budget for this particular service

Circuit breaker Request Service

Open Closed

Half open

Periodic sample

Fail fast

Figure 2.12 A circuit breaker in electrical engineering protects a circuit from being destroyed by a current that is too high. The software equivalent does the same thing for a service that would otherwise be overwhelmed by too many requests. 30 CHAPTER 2 A walk-through of the Reactive Manifesto

call, then the circuit breaker trips; from then on, requests will take a different route of processing that either fails fast or gives degraded service, just as in the case of over- flowing the bounded queue in front of the service. The same should also happen if the service replies with failures repeatedly, because then it is not worth the effort to send requests. This not only benefits the user by insulating it from the faulty service but also has the effect of reducing the load on the struggling service, giving it some time to recover and empty its queues. It would also be possible to monitor such occurrences and reinforce the resources for the overwhelmed service in response to the increased load. When the service has had some time to recuperate, the circuit breaker should snap back into a half-closed state in which some requests are sent in order to test whether the service is back in shape. If not, then the circuit breaker can trip again immediately; otherwise, it closes automatically and resumes normal operations. The Circuit Breaker pattern is discussed in detail in chapter 12.

2.4.3 Supervision In section 2.2, a simple function call returned a result synchronously:

val result = f(42)

In the context of a larger program, an invocation of f might be wrapped in an exception handler for reasonable error conditions, such as invalid input leading to a divide- by-zero error. Implementation details can result in exceptions that are not related to the input values. For example, a recursive implementation might lead to a stack overflow, or a distributed implementation might lead to networking errors. There is little the user of the service can do in those cases:

try { f(i) }catch{ Reasonable case ex: java.lang.ArithmeticException => Int.MaxValue response case ex: java.lang.StackOverflowError => ??? case ex: java.net.ConnectException => ??? Now what? }

Responses—including validation errors—are communicated back to the user of a service, whereas failures must be handled by the one who operates the service. The term that describes this relationship in a computer system is supervision. The supervisor is responsible for keeping the service alive and running. Figure 2.13 depicts these two different flows of information. The service internally handles everything it knows how to handle; it performs validation and processes requests, but any exceptions it cannot handle are escalated to the supervisor. While the service is in a broken state, it cannot process incoming requests. Imagine, for example, a service that depends on a working database connection. When the connection breaks, the database driver will throw an exception. If you tried to handle this Losing strong consistency 31

Failure

Figure 2.13 Supervision means that normal requests and responses (including negative ones Request such as validation errors) flow separately from failures: while the former are exchanged between the user and the service, the latter travel from the service to its supervisor.

Response

case directly within the service by attempting to establish a new connection, then that logic would be mixed with all the normal business logic of this service. But, worse, this service would need to think about the big picture as well. How many reconnection attempts make sense? How long should it wait between attempts? Handing those decisions off to a dedicated supervisor allows the separation of concerns—business logic versus specialized fault handling—and factoring them out into an external entity also enables the implementation of an overarching strategy for several supervised services. The supervisor could, for example, monitor how frequently failures occur on the primary database back-end system and fail over to a sec- ondary database replica when appropriate. In order to do that, the supervisor must have the power to start, stop, and restart the services it supervises: it is responsible for their lifecycle. The first system that directly supported this concept was Erlang/OTP, implementing the Actor model (discussed in chapter 3). Patterns related to supervision are described in chapter 12. 2.5 Losing strong consistency One of the most famous theoretical results on distributed systems is Eric Brewer’s CAP theorem,9 which states that any networked shared-data system can have at most two of three desirable properties: Consistency (C) equivalent to having a single up-to-date copy of the data High availability (A) of that data (for updates) Tolerance to network partitions (P) This means that during a network partition, at least one of consistency and availability must be sacrificed. If modifications continue during a partition, then inconsistencies can occur. The only way to avoid that would be to not accept modifications and thereby be unavailable. As an example, consider two users editing a shared text document using a service like Google Docs. Hopefully, the document is stored in at least two different locations

9 S. Gilbert and N. Lynch, “Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services,” ACM SIGACT News 33, no. 2 (2002), 51-59, http://dl.acm.org/citation.cfm?id=564601. 32 CHAPTER 2 A walk-through of the Reactive Manifesto

in order to survive a hardware failure of one of them, and both users randomly connect to some replica to make their changes. Normally, the changes will propagate between them, and each user will see the other’s edits; but if the network link between the replicas breaks down while everything else keeps working, both users will continue editing and see their own changes but not the changes made by the other. If both replace the same word with different improvements, then the result will be that the document is in an inconsistent state that needs to be repaired when the network link starts working again. The alternative would be to detect the network failure and forbid further changes until it is working again—leading to two unhappy users who not only will be unable to make conflicting changes but also will also be prevented from working on completely unrelated parts of the document. Traditional data stores are relational databases that provide a very high level of consistency guarantees, and customers of database vendors are accustomed to that mode of operation—not least because a lot of effort and research has gone into making databases efficient in spite of having to provide ACID10 transaction semantics. For this reason, distributed systems have so far concentrated critical components in a way that provides strong consistency. In the example of two users editing a shared document, a corresponding strongly consistent solution would mean that every change—every keypress—would need to be confirmed by the central server before being displayed locally, because otherwise one user’s screen could show a state that was inconsistent with what the other user saw. This obviously does not work, because it would be irritating to have such high latency while typing text—we are used to characters appearing instantly. This solution would also be costly to scale up to millions of users, considering the high-availability setups with log replication and the license fees for the big iron database. Compelling as this use case may be, Reactive systems present a challenging architecture change: the principles of resilience, scalability, and responsiveness need to be applied to all parts of the system in order to obtain the desired benefits, eliminating the strong transactional guarantees on which traditional systems were built. Eventu- ally, this change will have to occur, though—if not for the benefits outlined in the previous sections, then for physical reasons. The notion of ACID transactions aims at defining a global order of transactions such that no observer can detect inconsistencies. Taking a step back from the abstractions of programming into the physical world, Einstein’s theory of relativity has the astonishing property that some events cannot be ordered with respect to each other: if even a ray of light cannot travel from the location of the first event to the location of the second before that event happens, then the observed order of the two events depends on how fast an observer moves relative to those locations. Although we do not yet need to worry about computers traveling near the speed of light with respect to each other, we do need to worry about the speed of light between

10 Atomicity, consistency, isolation, durability. Losing strong consistency 33

even computers that are stationary. Events that cannot be connected by a ray of light as just described cannot have a causal order between them. Limiting the interactions between systems to proceed, at most, at the speed of light would be a solution to avoid ambiguities, but this is becoming a painful restriction already in today’s processor designs: agreeing on the current clock tick on both ends of a silicon chip is one of the limiting factors when trying to increase the clock frequency.

2.5.1 ACID 2.0 Systems with an inherently distributed design are built on a different set of principles. One such set is called BASE: Basically available Soft state (state needs to be actively maintained instead of persisting by default) Eventually consistent The last point means that modifications to the data need time to travel between distributed replicas, and during this time it is possible for external observers to see data that are inconsistent. The qualification “eventually” means the time window during which inconsistency can be observed after a change is bounded; when the system does not receive modifications any longer and enters a quiescent state, it will eventually become fully consistent again. In the example of editing a shared document, this means although you see your own changes immediately, you might see the other’s changes with some delay; and if conflicting changes are made, then the intermediate states seen by both users may be different. But once the incoming streams of changes end, both views will eventually settle into the same state for both users. In a note11 written 12 years after the CAP conjecture, Eric Brewer remarks thus: This [see above] expression of CAP served its purpose, which was to open the minds of designers to a wider range of systems and tradeoffs; indeed, in the past decade, a vast range of new systems has emerged, as well as much debate on the relative merits of consistency and availability. The “2 of 3” formulation was always misleading because it tended to oversimplify the tensions among properties. Now such nuances matter. CAP prohibits only a tiny part of the design space: perfect availability and consistency in the presence of partitions, which are rare. In the argument involving Einstein’s theory of relativity, the time window during which events cannot be ordered is very short—the speed of light is rather fast for everyday observations. In the same spirit, the inconsistency observed in eventually consistent systems is also short-lived; the delay between changes being made by one user and being visible to others is on the order of tens or maybe hundreds of milliseconds, which is good enough for collaborative document editing.

11 Eric Brewer, “CAP Twelve Years Later: How the ‘Rules’ Have Changed,” InfoQ, May 30, 2012, https://www .infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed. 34 CHAPTER 2 A walk-through of the Reactive Manifesto

BASE has served as an important step in evolving our understanding of which properties are useful and which are unattainable, but as a definitive term it is too imprecise. Another proposal brought forward by Pat Helland at React Conf 2014 is ACID 2.0: Associative Commutative Idempotent Distributed The last point just completes the familiar acronym, but the first three describe underlying mathematical principles that allow operations to be performed in a form that is eventually consistent by definition: if every action is represented such that it can be applied in batches (associative) and in any order (commutative) and such that applying it multiple times is not harmful (idempotent), then the end result does not depend on which replica accepts the change and in which order the updates are disseminated across the network—even resending is fine if reception is not yet acknowledged. Other authors, such as Peter Bailis and Martin Kleppmann, are pushing the envelope of how far we can extend consistency guarantees without running into the forbid- den spot of the CAP theorem: with the help of tracking the causality relationship between different updates, it seems possible to get very close to ACID semantics while minimizing the sacrifice in terms of availability. It will be interesting to see where this field of research will be in 10 years.

2.5.2 Accepting updates Only during a network partition is it problematic to accept modifications on both disconnected sides, although even for this case solutions are emerging in the form of conflict-free replicated data types (CRDTs). These have the property of merging cleanly when the partition ends, regardless of the modifications that were done on either side. Google Docs employs a similar technique called operational transformation.12 In the scenario in which replicas of a document get out of sync due to a network partition, local changes are still accepted and stored as operations. When the network connection is back in working condition, the different chains of operations are merged by bringing them into a linearized sequence. This is done by rebasing one chain on top of the other so that instead of operating on the last synchronized state, the one chain is transformed to operate on the state that results from applying the other chain before it. This resolves conflicting changes in a deterministic way, leading to a consistent document for both users after the partition has healed. Data types with these nice properties come with certain restrictions in terms of which operations they can support. There will naturally be problems that cannot be

12 David Wang, Alex Mah, and Soren Lassen, “Google Wave Operational Transformation,” July 2010, http://mng.bz/Bry5. The need for Reactive design patterns 35

stated using them, in which case you have no choice but to concentrate these data in one location only and forgo distribution. But our intuition is that necessity will drive the reduction of these issues by researching alternative models for the respective problem domain, forming a compromise between the need to provide responsive services that are always available and the business-level desire for strong consistency. One example from the real world is automated teller machines (ATMs): bank accounts are the traditional example of strong transactional reasoning, but the mechanical implementation of dispensing cash to account owners has been eventually consistent for a long time. When you go to an ATM to withdraw cash, you would be annoyed with your bank if the ATM did not work, especially if you needed the money to buy that anniversary present for your spouse. Network problems do occur frequently, and if the ATM rejected customers during such periods, that would lead to lots of unhappy customers—we know that bad stories spread a lot easier than stories that say “It just worked as it was supposed to.” The solution is to still offer service to the customer even if certain features like overdraft protection cannot work at the time. You might, for example, get less cash than you wanted while the machine cannot verify that your account has sufficient funds, but you would still get some bills instead of a dire “Out of Service” error. For the bank, this means your account may be overdrawn, but chances are that most people who want to withdraw money have enough to cover the transaction. And if the account has turned into a mini loan, there are established means to fix that: society provides a judicial system to enforce those parts of the contract that the machine could not, and in addition the bank charges fees and earns interest as long as the account holder owes it money. This example highlights that computer systems do not have to solve all the issues around a business process in all cases, especially when the cost of doing so would be prohibitive. It can also be seen as a system that falls back to an approximate solution until its nominal functionality can be restored. 2.6 The need for Reactive design patterns Many of the discussed solutions and most of the underlying problems are not new. Decoupling the design of different components of a program has been the goal of computer science research since its inception, and it has been part of the common literature since the famous 1994 Design Patterns book.13 As computers became more and more ubiquitous in our daily lives, programming moved accordingly into the focus of society and changed from an art practiced by academics and later by young “fanatics” in their basements to a widely applied craft. The growth in sheer size of computer systems deployed over the past two decades led to the formalization of designs building on top of the established best practices and widening the scope of what we consider

13 Erich Gamma et al., Design Patterns: Elements of Reusable Object-Oriented Software, Addison-Wesley Professional (1994). 36 CHAPTER 2 A walk-through of the Reactive Manifesto

charted territory. In 2003, Enterprise Integration Patterns14 covered message passing between networked components, defining communication and message-handling patterns—for example, implemented by the Apache Camel project. The next step was called service-oriented architecture (SOA). While reading this chapter, you will have recognized elements of earlier stages, such as the focus on message passing and services. The question naturally arises, what does this book add that has not already been described sufficiently elsewhere? Espe- cially interesting is a comparison to the definition of SOA in Arnon Rotem-Gal-Oz’s SOA Patterns (Manning, 2012): DEFINITION: Service-oriented architecture (SOA) is an architectural style for building systems based on interactions of loosely coupled, coarse-grained, and autonomous components called services. Each service exposes processes and behavior through contracts, which are composed of messages at discoverable addresses called endpoints. A service’s behavior is governed by policies that are external to the service itself. The contracts and messages are used by external components called service consumers. This definition focuses on the high-level architecture of an application, which is made explicit by demanding that the service structure be coarse-grained. The reason for this is that SOA approaches the topic from the perspective of business requirements and abstract software design, which without doubt is very useful. But as we have argued, technical reasons push the coarseness of services down to finer levels and demand that abstractions like synchronous blocking network communication be replaced by explicitly modeling the message-driven nature of the underlying system.

2.6.1 Managing complexity Lifting the level of abstraction has proven to be the most effective measure for increasing the productivity of programmers. Exposing more of the underlying details seems like a step backward on this count, because abstraction is usually meant to hide com- plications from view. This consideration neglects the fact that there are two kinds of complexity: Essential complexity is the kind that is inherent in the problem domain. Incidental complexity is the kind that is introduced solely by the solution. Coming back to the example of using a traditional database with transactions as the backing store for a shared document editor, the ACID solution tries to hide the essential complexity present in the domain of networked computer systems, introducing incidental complexity by requiring the developer to try to work around the performance and scalability issues that arise.

14 Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, Addison-Wesley Professional (2003). The need for Reactive design patterns 37

A proper solution exposes all the essential complexity of the problem domain, making it accessible to be tackled as is appropriate for the concrete use case, and avoids burdening the user with incidental complexity that results from a mismatch between the chosen abstraction and the underlying mechanics. This means that as your understanding of the problem domain evolves—for example, recognizing the need for distribution of computation at much finer granularity than before—you need to keep reevaluating the existing abstractions in view of whether they capture the essential complexity and how much incidental complexity they add. The result will be an adaptation of solutions, sometimes representing a shift in which properties you want to abstract over and which you want to expose. Reactive service design is one such shift, which makes some patterns like synchronous, strongly consistent service coupling obsolete. The corresponding loss in level of abstraction is countered by defining new abstractions and patterns for solutions, akin to restacking the building blocks on top of a realigned foundation. The new foundation is message orientation, and in order to compose large-scale applications on top of it, you need suitable tools to work with. The patterns discussed in the third part of this book are a combination of well-worn, comfortable instruments like the Circuit Breaker pattern as well as emerging patterns learned from wider usage of the Actor model. But a pattern consists of more than a description of a prototypical solution; more important, it is characterized by the problem it tries to solve. The main contribution of this book is therefore to discuss Reactive design patterns in light of the four tenets of the Reactive Manifesto.

2.6.2 Bringing programming models closer to the real world Our final remark on the consequences of Reactive programming takes up the strands that shone through in several places already. You have seen that the desire to create self-contained pieces of software that deliver service to their users reliably and quickly led to a design that builds on encapsulated, independently executed units of computation. The compartments between the bulkheads form private spaces for services that communicate only using messages in a high-level messaging language. These design constraints are familiar from the physical world and from our society: humans also collaborate on larger tasks, perform individual tasks autonomously, communicate via high-level language, and so on. This allows us to visualize abstract software concepts using well-known, customary images. We can tackle the architecture of an application by asking, “How would you do it given a group of people?” Software development is an extremely young discipline compared to the organization of labor between humans over the past millennia, and by using the knowledge we have built up, we have an easier time breaking down systems in ways that are compatible with the nature of distributed, autonomous implementation.

38 CHAPTER 2 A walk-through of the Reactive Manifesto

Of course, we should stay away from abuses of anthropomorphism: we are slowly eliminating terminology like “master/slave” in recognition that not everybody takes the technical context into account when interpreting them.15 But even responsible use offers plentiful opportunities for spicing up possibly dull work a little: for example, by calling a component responsible for writing logs to disk a Scribe. Implement- ing that class will have the feel of creating a little robot that will do certain things you tell it to and with which you can play a bit—others call that activity writing tests and make a sour face while saying so. With Reactive programming, you can turn this around and realize: it’s fun! 2.7 Summary This chapter laid the foundation for the rest of the book, introducing the tenets of the Reactive Manifesto: Responsive Resilient Elastic Message-driven We have shown how the need to stay responsive in the face of component failure defines resilience, and likewise how the desire to withstand surges in the incoming load elucidates the meaning of scalability. Throughout this discussion, you have seen the common theme of message orientation as an enabler for meeting the other three challenges. In the next chapter, we will introduce the tools of the trade: event loops, Futures and Promises, Reactive Extensions, and the Actor model. All these make use of the functional programming paradigm, which we will look at first.

15 Although terminology offers many interesting side notes: for example, a client is someone who obeys (from the Latin cluere), whereas server derives from slave (from the Latin servus)—so a client–server relationship is somewhat strange when interpreted literally. An example of naming that can easily prompt out-of-context interpretation is a hypothetical method name like harvest_dead_children(). In the interest of reducing nontechnical arguments about code, it is best to avoid such terms. Tools of the trade

The previous chapter explained why you need to be Reactive. Now we will turn our attention to the question of how you can achieve this goal. In this chapter, you will learn: The earliest Reactive approaches Important functional programming techniques Strengths and weaknesses of the existing Reactive tools and libraries 3.1 Early Reactive solutions Over the past 30 years, people have designed many tools and paradigms to help build Reactive applications. One of the oldest and most notable is the Erlang programming language (www.erlang.org), created by Joe Armstrong and his team at Ericsson in the mid-1980s. Erlang was the first language that brought Actors, described later in this chapter, into mainstream popularity. Armstrong and his team faced a daunting challenge: to build a language that would support the creation of distributed applications that are nearly impervious to failure. Over time, Erlang evolved in the Ericsson laboratory, culminating with its use in the late 1990s to build the AXD 301 telephone switch, which reportedly achieved “nine nines” of uptime—availability 99.9999999% of the time. Consider exactly what that means. For a single application running on a single machine, that would be roughly 3 seconds of downtime in 100 years!

100 years *365days/year *24hours/day *60minutes/day *60seconds/minute =3,153,600,000seconds

39 40 CHAPTER 3 Tools of the trade

3,153,600,000 seconds *0.000000001expecteddowntime =3.1536secondsofdowntimein100years

Of course, such long-lasting, near-perfect uptime is purely theoretical; modern computers haven’t even been around 100 years. The study upon which this claim was based was performed by British Telecom in 2002–2003 and involved 14 nodes and a calculation based on 5 node-years of study.1 Such approximations of downtime depend as much on the hardware as on the application itself, because unreliable computers put an upper limit on the availability of even the most resilient software. But such theoretical uptime illustrates the extraordinary fault tolerance possible in a Reac- tive application. Amazingly, no other language or platform has made claims compara- ble to Erlang’s. Erlang employs a dynamic type system and ubiquitous pattern matching to capture the dynamic nature of Actor systems, and it copies message data for every message it passes between Actors. The data has to be copied as shown in figure 3.1 because there is no shared heap space between two Actor processes in the BEAM VM, the virtual machine on which Erlang runs. Data sent between Actors must be copied into the receiving Actor process’s heap prior to sending the message, to guarantee isolation of Actors and to prevent concurrent access to the data. Although these features provide additional safety, ensuring that any Erlang Actor can receive any message and no data can be shared, they lower the application’s potential throughput. On the other hand, garbage collection can be performed independently for all process heaps2 and thus completes more quickly and with predict- able latency. All this copying would not be necessary if the Actors all shared the same heap. Then, two Actors could share a pointer to the same message. For this to work, though, one critical condition must be met: the data cannot be allowed to change. Functional programming addresses this challenge.

Sending process heap Receiving process heap

Data to Data send received

Msg. Msg. data Figure 3.1 In Erlang, data Sending Receiving in a sending Actor’s heap is actor actor transferred via a message to the heap of a receiving Actor.

1 Mats Cronqvist, “The Nine Nines,” 2012, www.erlang-factory.com/upload/presentations/243/ErlangFactory SFBay2010-MatsCronqvist.pdf. 2 One exception are binary strings longer than 64 bytes, which are stored in a separate heap and managed by reference counting. Functional programming 41

3.2 Functional programming The concepts of functional programming have been around for a very long time, but only recently have they gained favor in mainstream programming languages. Why did functional programming disappear from the mainstream for so long, and why is its popularity surging now? The period between 1995 and 2008 was essentially the Dark Ages of functional programming, as languages such as C, C++, and Java grew in usage, and the imperative, object-oriented programming style became the most popular way to write applications and solve problems. The advent of multiple cores opened up new opportunities for parallelization, but imperative programming constructs with side effects can be difficult to reason about in such an environment. Imagine a C or C++ developer who already has the burden of managing their own memory usage in a single-threaded application. In a multicore world, they must now manage memory across multiple threads while also trying to figure out who can access shared mutable memory at what time. This makes a job that was hard to do and verify in the first place into something that is daunting for even the most senior C/C++ developers. This has led to a veritable Renaissance in functional programming. Many languages now include constructs from functional programming, because these better support reasoning about problems in concurrent and parallelized applications. Code written in a functional style makes it easier for developers to reason about what the application is doing at any given point in time. The core concepts of functional programming have been around for many years: they were first defined in the lambda calculus by Alonzo Church in the 1930s.3 The essence of functional programming is the insight that programs can be written in terms of pure mathematical functions: that is, functions that return the same value every time they are passed the same inputs and that cause no side effects. Writing code in functional programming is analogous to composing functions in mathematics. With tools for functional programming now at our disposal, it is truly a wonderful time to be a programmer again, because we can solve problems with languages that support the functional, the side-effecting, or both paradigms simultaneously. Next, we will examine some core concepts of functional programming that go beyond pure function composition: immutability, referential transparency, side effects, and functions as first-class citizens.

3.2.1 Immutability A variable is said to have mutable state when it can refer to different values at different times. In an otherwise purely functional program, mutable state is called impure and is considered dangerous. Mutability is represented by any variable or field that is not stable or final and can be changed or updated while the application is running; some

3 Alonzo Church, “The Calculi of Lambda-Conversion,” Annals of Mathematical Studies 6, Princeton University Press (1941). 42 CHAPTER 3 Tools of the trade

examples are shown in listing 3.1. When you use a final, immutable variable, you can reason more easily about what value it will have at a given time because you know that nothing can possibly change it after it is defined. This applies to data structures as well as to simple variables: any action performed on an immutable data structure results in the creation of a new data structure, which holds the result of the change. The original data structure remains unchanged. Therefore, any other part of the program that continues to use the original data structure does not see the change. By using immutability throughout an application, you can limit the places where mutation is possible to a very small section of code. So, the possibility of contention, where multiple threads attempt to access the same resource at the same time and some are forced to wait their turn, is limited in scope to a small region. Contention is one of the biggest drains on performance in code that runs on multiple CPU cores; it should be avoided where possible. The astute reader will notice the tension between funneling all changes through a single point and trying to avoid contention. The key to solving this apparent paradox is that having all code that performs mutation within a small scope makes contention easier to manage. With full oversight of the problem, you can tune the behavior: for example, by dividing a complex state representation into several mutable variables that can usually be updated independently—hence, without contention.

Listing 3.1 Unsafe, mutable message class, which may hide unexpected behavior

import java.util.Date; Fields should be final public class Unsafe { so the compiler can private Date timestamp; enforce immutability. private final StringBuffer message; Mutable because the public Unsafe(Date timestamp, StringBuffer message) { content of the buffer this.timestamp = timestamp; is not stable this.message = message; }

public synchronized Date getTimestamp() { return timestamp; }

public synchronized void setTimestamp(Date timestamp) { this.timestamp = timestamp; } Mutable because the timestamp can be Mutable public StringBuffer getMessage() { changed via the setter. Synchronizing because the return message; both the getter and setter adds content of } thread safety but does nothing the buffer is } to relieve contention between not stable threads accessing the object.

It is better to enforce immutability using the compiler rather than convention. This implies passing values to the constructor rather than calling setters, and using language features such as final in Java and val in Scala. Sometimes that is not possible, such as when an API requires an object to be created before all of its member values Functional programming 43

are known. In those situations, you may have to resort to initialization flags to prevent values from being set more than once or after the object is already in use. Immutable data structures like the one shown in listing 3.2 ensure that the values returned by an object do not change. It does little good to ensure that the variable holding a Date is not reassigned if its content can be changed. The problem is not thread safety. The problem is that mutable state makes it far more difficult to reason about what the code is doing. There are several alternatives: The first choice is to use an intrinsically immutable data structure. Some languages provide extensive libraries of immutable collection implementations, or you might incorporate a third-party library. You can write a wrapper class to block access to the mutating methods. Ensure that no references remain to the backing mutable data structure once it is initialized. They can inadvertently defeat the purpose of the wrapper. Copy-on-read semantics creates and returns a complete copy of the data structure every time it is read from the object. This ensures that readers do not have access to the original object, but it can be expensive. As with immutable wrap- pers, you must ensure that no outside references remain to the still-writable data structure within the object. Copy-on-write semantics creates and returns a complete copy of the data structure whenever it is modified and ensures that users of the object cannot modify it through references they received from accessor methods. This prevents callers from changing the object’s underlying, mutable data structure, and it leaves previously acquired reader references unchanged. The data structure can block use of the mutators once the data structure is initialized. This typically requires adding a flag to mark the data structure as read- only after it has been initialized.

Listing 3.2 Immutable message class that behaves predictably and is easier to reason about

import java.util.Date; All fields are final It is good and contain stable practice to public class Immutable { data structures. declare the private final Date timestamp; parameters private final String message; final, too. public Immutable(final Date timestamp, final String message) { this.timestamp = new Date(timestamp.getTime()); this.message = message; Ensures that the } timestamp cannot be changed by public Date getTimestamp() { other code Getters but return new Date(timestamp.getTime()); no setters } public String getMessage() { return message; } }} 44 CHAPTER 3 Tools of the trade

Java does not make everything immutable by default, but liberal use of the final key- word is helpful. In Scala, case classes provide immutability by default as well as additional, very convenient features such as correct equality and hash-code functions:

Even with case classes, import java.util.Date you must take care that the fields hold immutable data structures—using case class Message(timestamp: Date, message: String) the Date’s setters will break this.

3.2.2 Referential transparency An expression is said to be referentially transparent if replacing it with a single value (a constant) has no impact on the execution of the program.4 So, evaluating a referentially transparent expression—that is, performing an operation on some data—has no impact on that data, and no side effects can occur. For example, the act of adding, removing, or updating a value in an immutable list results in a new list being created with the changed values; any part of the program still observing the original list sees no change. Consider Java’s java.lang.StringBuffer class. If you call the reverse method on a StringBuffer, you will get a reference to a StringBuffer with the values reversed. But the original StringBuffer reference refers to the same instance and therefore now also has changed its value:

final StringBuffer original = new StringBuffer(“foo”); final StringBuffer reversed = myStringBuffer.reverse(); System.out.println(String.format( "original '%s', new value '%s'", original, reversed)); Result: original 'oof', new value 'oof'

This is an example of referential opacity: the value of the expression myStringBuffer .reverse() changes when it is evaluated. It cannot be replaced by its result without altering the way the program executes. The java.lang.StringBuilder class has the same problem. Note, however, that a function call can be referentially transparent even if the function modifies internal state, as long as the function call can be replaced with its result without affecting the program’s output. For example, if an operation is likely to be performed multiple times, internally caching the result the first time can speed up execution without violating referential transparency. An example of this approach is shown in the next listing.

4 http://en.wikipedia.org/wiki/Referential_transparency_(computer_science) Functional programming 45

Listing 3.3 Referential transparency: allowing substitution of precomputed values

public class Rooter { private final double value; Mutable field for caching private Double root = null; a computed value

public Rooter(double value) { this.value = value; }

public double getValue() { return value; }

public double getRoot() { if (root == null) { root = Math.sqrt(value); } Mutability is never return root; observable outside. } }

3.2.3 Side effects Side effects are actions that break the referential transparency of later method calls by changing their environment such that the same inputs now lead to different results. Examples are modifying some system property, writing log output to the console, and sending data over a network. Pure functions have no side effects. Some functional programming languages such as Haskell enforce rules about where side effects can exist and when they can take place—a great help when reasoning about the correctness of code. Side effects matter because they limit how an object can be used. Consider the following class.

Listing 3.4 Limiting usability with side effects

import java.io.Serializable;

public class SideEffecting implements Serializable, Cloneable {

private int count;

public SideEffecting(int start) { this.count = start; }

public int next() { this.count += Math.incrementExact(this.count); return this.count; } } 46 CHAPTER 3 Tools of the trade

Every call to next() will return a different value. Consequently, the result of something like the example in listing 3.5 can give you a very unpleasant experience:

final int next = se.next(); if (logger.isDebugEnabled()) { Probably meant to logger.debug("Next is " ++ se.next()); reference the variable! } return next;

Even worse, something like new SideEffecting(Integer.MAX_VALUE - 2) will cause the side effect after a few calls to become an ArithmeticException. Sometimes side effects are more subtle. Suppose the object needs to be passed remotely. If it is immutable and without side effects, it can be serialized and reconsti- tuted on the remote system, which then will have its own identical and unchanging copy. If there are side effects, the two copies will diverge from each other. This is especially problematic when the original system was not envisioned to have distributed operations. You may blithely assume that updates are being applied to the one and only instance of an object, without realizing the trouble that will cause when scalability requirements lead to copies of the object being kept on multiple servers.

3.2.4 Functions as first-class citizens In a language where functions are first-class citizens, a function is a value just like an Integer or a String and can be passed as an argument to another function or method. The idea of doing so is to make code more composable: a function that takes a function as an argument can compose the calculation performed by its argument with calculations of its own, as illustrated by the call to .map in the following snippet:

final List numbers = Arrays.asList(1, 2, 3); final List numbersPlusOne = numbers.stream() Passes a function that .map(number -> number + 1) increments its argument by 1 .collect(Collectors.toList());

Many languages that are otherwise not supportive of functional programming have functions as first-class citizens, including JavaScript and Java 8. In the previous example, the function passed as an argument is a lambda expression. It has no name and exists only within the context of its call site. Languages that support functions as first-class citizens also allow you to assign this function to a named variable as a function value and then refer to it from wherever you see fit. In Python, you could do this like so:

>>> def addOne(x): ... returnx+1 ... >>> myFunction = addOne >>> myFunction(3) 4 Responsiveness to users 47

3.3 Responsiveness to users Beyond functional programming, to build Reactive applications you also need to use tools that give you responsiveness. This is not responsive web design,5 as it is known in the user-experience world, where a front end is said to be “responsive” if it resizes itself appropriately for the user’s viewing device. Responsiveness in Reactive applications means being able to quickly respond to user requests in the face of failure that can occur anywhere inside or outside the application. The performance trade-offs of Reactive applications are defined by the axiom that you can choose any two of the following three characteristics: High throughput Low latency, but also smooth and not jittery Small footprint These will all be defined further along in this section.

3.3.1 Prioritizing the three performance characteristics When you make architecture choices for a Reactive application, you are in essence giving priority to two of those three characteristics and choosing to sacrifice the remain- ing characteristic where necessary. This is not a law or theorem, but more of a guiding principle that is likely to be true in most cases. To get a very fast application with smooth, low latency, you typically have to give it more resources (footprint). An example of this is a high-performance messaging library written in Java, known as the Disrup- tor (http://lmax-exchange.github.io/disruptor). To get its tremendous throughput and smooth latency, the Disruptor has to preallocate all the memory it will ever use for its internal ring buffer, in order to prevent allocations at runtime that could lead to stop-the-world garbage collections and compaction pauses in the Java virtual machine. The Disruptor gains throughput by pinning itself to a specific execution core, thereby avoiding the cost of context switches6 between threads being swapped in and out on that core. This is another aspect of application footprint: there is now one fewer execution core available for use by other threads on that computer. The Storm framework (https://github.com/nathanmarz/storm), created by Nathan Marz and released in 2011 to much acclaim, provides capabilities for distributed processing of streaming data. Storm was created at Marz’s startup, BackType, which was purchased by Twitter and became the basis for its real-time analytics applications. But the implementation was not particularly fast, because it was built using Clojure and pure functional programming constructs. When Marz released version 0.7 in 2012, he used the Disruptor to increase throughput by as much as three times, at the cost of footprint. This trade-off matters to those who choose to deploy an application that uses Storm, particularly in the cloud, where one core on the VM must be

5 http://en.wikipedia.org/wiki/Responsive_web_design 6 http://en.wikipedia.org/wiki/Context_switch 48 CHAPTER 3 Tools of the trade

devoted solely to the Disruptor to maintain its speed. Note that the number of cores available to an application in a virtual environment is not an absolute value. As Doug Lea,7 the author of many concurrency libraries on the JVM, such as the ForkJoinPool and CompletableFuture, has said, “Ask your hypervisor how much memory you have, or how many cores you have. It’ll lie. Every time. It’s designed to lie. You paid them money so it would lie.”8 Developers have to take these variables into account when considering footprint in a cloud deployment. Some platforms have constraints that limit their ability to make these trade-offs. For example, a mobile application typically cannot give up footprint to increase throughput and make latency smoother, because a mobile platform typically has very limited resources. Imagine the cost of pinning a core on a mobile phone, both in terms of reduced resource availability for other applications and the phone’s operating system. A mobile phone has limited memory as well as constraints on power usage: you don’t want to drain the battery so quickly that no one would want to use your application. So, mobile applications typically attempt to minimize footprint while increasing throughput at the expense of latency, because users are more willing to accept latency on a mobile platform. Anyone who has used an application from a phone has experienced slowness due to network issues or packet loss. 3.4 Existing support for Reactive design Now that we have reviewed the basic concepts needed to understand and evaluate tools and language constructs for implementing Reactive design, it is time to look at the leading examples. Many languages have innovated in the area of dividing work and making it asynchronous. For each concept or implementation described in this section, we will provide an evaluation of how well it meets the tenets of the Reactive Manifesto. Note that although all are asynchronous and nonblocking in and of themselves, it is still up to you to ensure that the code you write remains asynchronous and nonblocking as well, in order to remain Reactive.

3.4.1 Green threads Some languages do not include built-in constructs for running multiple threads within the same operating system process. In these cases, it is still possible to implement asynchronous behavior via green threads: threads scheduled by the user process rather than the operating system.9 Green threads can be very efficient but are restricted to a single physical machine. It is impossible, without the aid of delimited portable continuations,10 to share the

7 http://en.wikipedia.org/wiki/Doug_Lea 8 Doug Lea, “Engineering Concurrent Library Components” (talk, Emerging Technologies Conference, 2013), http://chariotsolutions.com/screencast/phillyete-screencast-7-doug-lea-engineering-concurrent-library-components. 9 http://en.wikipedia.org/wiki/Green_threads 10 http://en.wikipedia.org/wiki/Delimited_continuation Existing support for Reactive design 49

processing of a thread across multiple physical machines. Delimited portable continuations allow you to mark points in logic where the execution stack can be wrapped up and exported either locally or to another machine. These continuations can then be treated as functions. This is a powerful idea, implemented in a Scheme library called Termite (https://code.google.com/p/termite). But green threads and continuations do not provide for resilience, because there is currently no way to supervise their execution; they are therefore lacking with respect to fault tolerance. Waldo et al. note11 that it is not a good idea to try to make logic that executes in a distributed context appear to be local. This postulate applies to green threads as well. If we take “local” to mean local to the thread, and “distributed/remote” to mean on another thread and thus asynchronous, you would not want “distributed/remote” to appear “local” because this would obscure the difference between synchronous and asynchronous operations. It would be hard to tell which operations can block the program’s progress and which cannot!

REACTIVE EVALUATION OF GREEN THREADS Green threads are asynchronous and nonblocking, but they do not support message passing. They do not scale up to use multiple cores on a machine by themselves, although if the runtime supports it, it is possible to have more than one in a process, or multiple processes can be run. They do not scale outward across nodes in a network. They also do not provide any mechanisms for fault tolerance, so it is up to developers who use them to write their own constructs to handle any failure that may occur.

3.4.2 Event loops When a language or platform does not support multiple threads in one process, you can still get asynchronous behavior by making green threads that share an event loop. This loop provides a mechanism for sharing a single execution thread among several logical threads at the same time. The idea is that although only a single thread can execute at a given moment, the application should not block on any operation and should instead make each thread yield until such time as the external work it needs, such as calling a data store, is completed. At that time, the application can invoke a callback to perform a predefined behavior. This is very powerful: Node.js (http://nodejs.org), for example, uses a single-threaded JavaScript runtime to perform considerably more work because it doesn’t have to wait for every operation to complete before handling other work. Event loops are most typically implemented with callbacks. This would be okay if only a single callback could be referenced at a time, but as an application’s functionality grows, this typically is not the case. The terms callback hell and pyramid of doom have been coined to represent the interwoven spaghetti code that often results from popular tools like Node.js. Furthermore, event loops based on a single-threaded process are viable

11 Jim Waldo, Geoff Wyant, Ann Wollrath, Sam Kendall: “A Note on Distributed Computing,” Sun Microsystems Laboratories, 1994, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.7628. 50 CHAPTER 3 Tools of the trade

only for uses that are I/O-bound or when the use case is specific to handling input and output. Trying to use an event loop for CPU-bound operations will defeat the advantage of this approach. Here is a simple example of a Node.js application. Note that running this server and using Google’s Chrome browser to send requests to the address 127.0.0.1:8888 may result in a doubling of the counter value on each request. Chrome has a known issue with sending an additional request for favicon.ico with every request:12

var http = require('http'); Sets up the server, with a var counter = 0; callback function applied to http.createServer(function (req, res) { respond to each request counter += 1; res.writeHead(200, {'Content-Type': 'text/plain'}); Informs the user res.end('Response: ' + counter+'viacallback\n'); where the server }).listen(8888, '127.0.0.1'); is reachable

console.log('Server up on 127.0.0.1:8888, send requests!');

REACTIVE EVALUATION OF EVENT LOOPS The suitability of an event loop for a Reactive application depends on the implementation. As deployed via Node.js in JavaScript, event loops are similar to green threads in that they are asynchronous and nonblocking but do not support message passing. They do not scale up to use multiple cores on a machine by themselves, although if the runtime supports it, it is possible to have more than one in a process, or multiple processes can be run. They do not scale outward across nodes in a network. They also do not provide any mechanisms for fault tolerance, so it is up to developers to write their own constructs to handle any failure that may occur. But there are alternative implementations, such as Vert.x (http://vertx.io), which runs on the JVM and has a feel similar to Node.js but supports multiple languages. Vert.x is a compelling solution because it provides a distributed approach to the event- loop model, using a distributed event bus to push messages between nodes. In a JVM deployment, it does not need to use green threads because it can use a pool of threads for multiple purposes. In this respect, Vert.x is asynchronous and nonblocking and does support message passing. It also scales up to use multiple cores, as well as scales out to use multiple nodes in a network. Vert.x does not have a supervision strategy for fault tolerance, but it is an excellent alternative to an event loop, particularly because it supports JavaScript just as Node.js does.

3.4.3 Communicating Sequential Processes Communicating Sequential Processes (CSP) is a mathematical abstraction of multiple processes, or threads in a single process, that communicate via message passing.13 You

12 Chrome issue report: https://code.google.com/p/chromium/issues/detail?id=39402. 13 C.A.R. Hoare, “Communicating Sequential Processes,” Communications of the ACM 21, no. 8 (1978): 666–677. Existing support for Reactive design 51 can define work to be performed concurrently in separate processes or threads, which then pass messages between them to share information. What makes CSP unique is that the two processes or threads do not have to know anything about one another, so they are nicely decoupled from the standpoint of sender and receiver but still coupled with respect to the value being passed. Rather than assuming that messages accumulate in queues until read, CSP uses rendezvous messaging: for a message to be passed, the sender and receiver must reach a point where the sender is ready to send and the receiver is ready to receive. Consequently, receipt of a message always synchronizes both processes. This is fundamentally different from Actors, which will be discussed in section 3.4.6. This also limits the two processes or threads in how distributed they can be, depending on how CSP is implemented. For example, CSP on the JVM as implemented by Clojure’s core.async library cannot be distributed across multiple JVM instances, even on the same machine. Neither can Go’s channels, also known as goroutines. Because CSP is defined formally and mathematically, it is theoretically provable whether a deadlock can or cannot occur inside of it, via a method called process analysis. Being able to statically verify the correctness of concurrency logic is a powerful idea. Note, however, that neither Clojure’s core.async nor Go’s channels have this capability; but if it is practical to implement, it would be very useful. Because no process or thread in a CSP-based application has to know about another, there is a form of location transparency: to write the code for one process or thread, you do not need to know about any other process or thread with which it will communicate. But the most popular implementations of CSP to date do not support communication between different nodes on a network, so they cannot support true location transparency. They also have difficulties with fault tolerance, because failure between two processes or threads cannot be managed easily. Instead, the logic in each process or thread must have the ability to manage any failure that could occur when communicating with the other side. Another potential downside is that nontrivial use of CSP can be difficult to reason about, because every process/thread can potentially interact with every other process/thread at each step. Here is a simple example of two communicating processes in Go. Interestingly, a Go function can create a channel and put values onto it as well as consume them, in effect stashing values off to the side for use later. In this example, one function produces messages and puts them onto the channel, and a second function consumes them: package main Imports required import ( language libraries "fmt" "time" ) func main() { iterations := 10 Creates a communication myChannel := make(chan int) channel for integer values 52 CHAPTER 3 Tools of the trade

go producer(myChannel, iterations) Starts asynchronous execution go consumer(myChannel, iterations) of producer and consumer

time.Sleep(500 * time.Millisecond) } func producer(myChannel chan int, iterations int) { for i := 1; i <= iterations; i++ { fmt.Println("Sending: ", i) myChannel <- i Sends a message to the channel } } func consumer(myChannel chan int, iterations int) { for i := 1; i <= iterations; i++ { recVal := <-myChannel Receives a message from the channel fmt.Println("Received: ", recVal) } }

REACTIVE EVALUATION OF CSP CSP is asynchronous and nonblocking and supports message passing in rendezvous fashion. It scales up to use multiple cores on a machine, but none of the current implementations scale outward across nodes. CSP does not provide any mechanisms for fault tolerance, so it is up to developers to write their own constructs to handle any failure that may occur.

3.4.4 Futures and promises A Future is a read-only handle to a value or failure that may become available at some point in time; a Promise is the corresponding write-once handle that allows the value to be provided. Note that these definitions are not universally established; the terminology chosen here is that used in C++, Java, and Scala.14 A function that returns its result asynchronously constructs a Promise, sets the asynchronous processing in motion, installs a completion callback that will eventually fulfill the Promise, and returns the Future associated with the Promise to the caller. The caller can then attach code to the Future, such as callbacks or transformations, to be executed when the Future’s value is provided. Normally, a function that returns a Future does not expose the underlying Promise to its caller. All Future implementations provide a mechanism to turn a code block—for example, a lambda expression—into a Future such that the code is dispatched to run on a different thread, and its return value fulfills the Future’s Promise when it becomes available. Futures therefore provide a simple way to make code asynchronous and implement parallelism. Futures return either the result of their successful evaluation or a representation of whatever error may have occurred during evaluation.

14 See https://en.wikipedia.org/wiki/Futures_and_promises for an overview. Existing support for Reactive design 53

Time Customer request

Check cache Not found! Cache

Check DB Data Found! Figure 3.2 Sequential lookup: store first, check the cache; then, if Customer response the data are not found, retrieve them from the database.

We turn now to an elegant example of Futures in practice: retrieval of data from multiple sources, where you prefer to access sources in parallel (simultaneously) rather than sequentially. Imagine a service that needs to return customer information that may be stored in a database somewhere far away but may also be cached in a store that is closer for performance reasons. To retrieve the data, the program should check the cache first to see whether it has the data needed to avoid an expensive database lookup. If there is a cache miss—if the information is not found—the program must look it up in the database. In a sequential lookup shown in figure 3.2, the calling thread tries to retrieve the data from the cache first. If the cache lookup fails, it then makes a call to the database and returns its response—at the cost of two lookups that took place one after the other. In a parallel lookup shown in figure 3.3, the calling thread sends requests to the cache and the database simultaneously. If the cache responds with a found customer record first, the response is sent back to the client immediately. When the database responds later with the same record, it is ignored. But if the cache lookup fails, the calling thread doesn’t have to make a subsequent database call, because that has already been done. When the database responds, the response is sent to the client right away, theoretically sooner than if the client had made sequential calls.

Time

Customer request Check DB

Check cache

Data Cache store

Not found! Figure 3.3 Parallel lookup: send Found! requests to the cache and the database simultaneously; the Customer response result is the first value returned. 54 CHAPTER 3 Tools of the trade

The code for parallel lookup may look similar to the following listing, written in Java 8 to exploit its nonblocking CompletableFuture class.

Listing 3.5 Retrieving the result from the faster source

public class ParallelRetrievalExample { final CacheRetriever cacheRetriever; final DBRetriever dbRetriever; ParallelRetrievalExample(CacheRetriever cacheRetriever, DBRetriever dbRetriever) { this.cacheRetriever = cacheRetriever; this.dbRetriever = dbRetriever; }

public Object retrieveCustomer(final long id) { final CompletableFuture