Everything You Need To Know About Queueing Theory October 26, 2015 • Revision 2 Copyright © 2015 VividCortex Meet the Author Baron Schwartz Baron is well-known for his contributions to the MySQL, PostgreSQL, and Oracle communities. Baron has helped build and optimize database systems for some of the largest Internet properties. He has written several books, including O'Reilly's best-selling High Performance MySQL. Prior to founding VividCortex, Baron managed consulting, support, training, and software engineering at Percona. He has a CS degree from the University of Virginia. Table of Contents • Queueing Theory is for Everyone 3 • What is Queueing Theory? 4 • Why Queueing Theory is Tricky 5 • Queueing Systems in Real Life 7 • Why Does Queueing Happen? 8 • The Queueing Theory Framework 10 • Describing Queueing Systems 15 • Calculating Queue Lengths and Wait Times 18 • The Erlang Queueing Formulas 19 • Approximations to Erlang Formulas 20 • Many Queues or One Combined Queue? 22 • Tradeoffs Between Cost and Quality of Service 24 • Applying Queueing Theory in the Real World 26 • Conclusions 28 • Further Reading 29 Copyright © 2015 VividCortex Page 3 Queueing Theory is for Everyone Whether you’re an entrepreneur, engineer, or manager, learning about queueing theory is one of the best ways to boost your performance. Queueing theory is fundamental to getting good return on your efforts. That’s because the results your systems and teams produce are heavily influenced by how much waiting takes place, and waiting is waste. Minimizing this waste is extremely important. It’s one of the biggest levers you will ever find for improving the cost and performance of your teams and systems. Unfortunately, queueing theory books are often terribly complicated and boring. But it doesn’t have to be that way! The concepts are easy to understand, and developing intuition about what happens in queues is the real low-hanging fruit. Plus, queueing theory is super-cool, and it’s a pity if you don’t learn about it. There are plenty of 350-page books about queueing theory, and lots of high-level introductions too. But these all either dive deep into pages full of equations and Greek symbols, or gloss over (or just outright lie about) the big, important truths of queueing. We need a middle ground. My hope is that this book can help you achieve the following: • Develop awareness of how queueing appears in everyday life. Queueing rules everything around you, but many people probably aren’t aware of it. • Build intuition of important concepts and truths that can guide you in decision making. • Help you see where your intuition is a trusty guide and where queueing will surprise you. • Help you understand the landscape of queueing theory from a general Copyright © 2015 VividCortex Page 4 framework, so you can quickly navigate more detailed treatises and find the right answers fast. Armed with the knowledge in this book, you will be able to quickly understand common scenarios such as: • If web traffic increases 30%, will the server get overloaded? • Should I buy faster disks, or should I just buy a few more of what I already have? • What if I specialize tasks in my team and create a handoff between roles? Knowing how to think about these kinds of questions will help you anticipate bottlenecks. As a result, you’ll build your systems and teams to be more efficient, more scalable, higher performance, lower cost, and ultimately provide better service both for yourself and for your customers. And when you need to calculate exactly how much things will bottleneck and how much performance will suffer, you’ll know how to do that too. As for scope, the aim of this book is to help you get answers for common scenarios and point you to the right resources for other cases. I’ll go into details where they help illuminate important concepts. I’ll also try to make it fun in a nerdy way. In the end, I hope the Aaron Neville and Linda Ronstadt song will take on new meaning for you: Don’t know much But I know I love queues And that may be All I need to know Copyright © 2015 VividCortex Page 5 What is Queueing Theory? Queueing theory is a broad field of study that focuses on the many nuances of how waiting lines behave. It’s a fascinating topic if you’re of a theoretical bent. But it’s highly relevant even if you’re the impatient “just answer my question!” type, too. Queueing theory boils down to answering simple questions like the following: • How likely is it that things will queue up and wait in line? • How long will the line be? • How long will the wait be? • How busy will the server/person/system servicing the line be? • How much capacity is needed to meet an expected level of demand? These questions can easily be rephrased and used to answer other questions, such as “how likely am I to lose business due to too-long waits or not enough capacity?” and “how much more demand can we satisfy without creating an unacceptably long wait?” And although I wrote the questions in simple terms, you can also answer sophisticated questions about the characteristics of the answers. For example, “how long will the wait be?” can be answered not only on average, but you can calculate things such as the variability of wait times, the distribution of wait times and how likely it is that someone will get extremely poor service, and so on. Copyright © 2015 VividCortex Page 6 Why Queueing Theory is Tricky Bad news: queueing theory is part of probability theory, and probability is hard and weird. Nothing leads you astray faster than trusting your intuition about probability. If your intuition worked, casinos would all go out of business, and your insurance rates would seem reasonable. In particular, human intuition is linear. Double the traffic, we think, and other things will double too. Wrong! Queueing systems are nonlinear. What’s worse is that they’re unpredictably nonlinear. Things seem to change gradually, until you reach some tipping point and boom, everything gets really bad, really fast. These transitions can be so sudden that they are very surprising. Our intuition isn’t geared to expect these disproportionate, sudden changes. The classic example you might recognize is the so-called hockey-stick curve. We need a picture to break up all this text, so I’ll introduce that now. This is the most famous queueing theory picture of all time: This curve shows how a system’s total response time increases as utilization increases. As utilization approaches 100%, response time goes to infinity. The exact shape of this curve depends on the characteristics Copyright © 2015 VividCortex Page 7 of the system and the work it’s doing, but the general principle applies to pretty much all systems. You’ll get familiar with nonlinearity over the course of this book. You might never be able to accurately estimate how much things will be nonlinear. But you’ll get pretty good at recognizing situations where someone needs to check whether bad stuff is going to happen. Then you’ll do a little bit of math or look things up in a table and find the answers you need. Queueing Systems in Real Life Most introductions to queueing theory talk about abstract things and use jargon a lot. They’ll usually discuss some real-life scenarios to help you understand the abstract things, but for some reason they always seem to pick bizarre situations I personally have never encountered1. Then they jump into discussions of Markovian something or other, and talk about births and deaths. At this point my eyes have usually glazed over. But life is full of queueing systems! • Your Kanban board • Getting a latte at the coffee shop • The line for the restroom after drinking the latte • Your computer’s CPUs and disks • At the traffic light • When the captain says you’re third in line for takeoff • Your load balancer • Pre-ordering the new iPhone 1 Often with British terminology, something about hiring lorries and abseiling Copyright © 2015 VividCortex Page 8 • Trying to pick the best line at the grocery checkout counter • Merging onto the freeway • Connection pooling Importantly, queueing systems are present in much more subtle scenarios too. Any process for handling units of work has queueing systems built into it. When you ask your graphic designer to produce a beautiful new ebook, for example (ahem), queueing is at work. One of the big wins is being able to recognize these systems, particularly in teams. Teams have communication structures, and communication—whether it’s email or chat at the water cooler—is queueing. Send an email to your colleague; when it’s in their inbox it’s queued. If a developer doesn’t know how to optimize a query and asks a DBA for help, and the DBA is busy, the developer’s request is queued. Design your organization with the recognition that teams are systems too, and you’ll have a chance to get much better results. Why Does Queueing Happen? Why do things wait in queues anyway? It might seem like a dumb question with an obvious answer. “Because there’s more work to do than capacity to do it, silly!” But that is not the right answer. Queueing happens even when there’s more than enough capacity to do the work. This is part of what’s really surprising and counterintuitive about queueing theory. This is so important that it needs to be emphasized: Queueing happens even when there’s more than enough capacity to do the work.