FACILITATING THE SPREAD OF KNOWLEDGE AND INNOVATION IN PROFESSIONAL DEVELOPMENT Advanced DevOps Toolchain eMag Issue 28 - May 2015

ARTICLE VIRTUAL PANEL ARTICLE Docker: Present Immutable Service and Future Infrastructure Discovery

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 1 Virtual Panel on Immutable Infrastructure Docker: Present and Future InfoQ reached out to experienced ops engineers to ask about Chris Swan presents an overview of the Docker journey so their definition and borders of immutable infrastructure as far and where it is headed along with its growing ecosystem well as its benefits and drawbacks, in particular when com- of tools for orchestration, composition and scaling. This ar- pared to “desired state” configuration management solutions. ticle provides both a business and a technical point of view Is it a step forward or backwards in effective infrastructure on Docker and separates the hype from the reality. management.

How I Built a Self-Service Automation Platform The current buzz in the industry is how DevOps is chang- ing the world and how the traditional role of operations is changing. Part of these discussions lead to building self-service automation tools which enable developers to actually deploy and support their own code. Brian Carpio describes his journey towards an infrastructure that development teams could deploy with the click of a mouse or an API call.

Managing Build Jobs for The number of jobs in a continuous inte- gration tool can range from a few to sev- eral thousand, all performing various functions. There is an approach to manage these jobs in a more efficient manner.

Service Discovery Building microservices using immutable infrastructure means that your services need to be reconfigured with the location of the other services they need to connect to. This process is called Service Dis- covery. This article compares the approaches by several key con- tenders, from Zookeeper to Consul, Etcd, Eureka and “rolling your own”.

FOLLOW US CONTACT US GENERAL FEEDBACK [email protected]

ADVERTISING [email protected]

EDITORIAL [email protected] facebook.com @InfoQ google.com linkedin.com /InfoQ /+InfoQ company/infoq

is InfoQ’s DevOps Lead Editor and an enthusiast MANUEL of Continuous Delivery and Agile practices. PAIS Manuel Pais tweets @manupaisable

A LETTER FROM and weaknesses of containers when compared to THE EDITOR the more established virtual machines. He also high- lights the expanding ecosystem around Docker, in- cluding tools and libraries for networking, composi- tion and orchestration of containers. Many companies haven taken the route of au- tomating delivery of traditional ops services such as provisioning infrastructure, deploying to production environments, adding metrics to monitor or creat- The edition follows in the footsteps of the previous ing test environments as similar as possible to pro- “DevOps Toolchain for Beginners” eMag. Once an duction. This decision often comes from reaching a organization has acquired the essential skill set to bottleneck in operations workload as development provision and manage infrastructure, set up delivery teams are empowered to deploy and own their ap- pipelines and analyze logs and metrics, it will then plications through their entire lifecycle in a DevOps gradually face more complex decisions. Decisions mindset. Brian Carpio describes his technical jour- that depend much more on the type of systems ar- ney into creating such a self-service platform. chitecture and organizational structure than on indi- When teams reap the benefits of continuous vidual tool choices. delivery, they want more of it. On every project. Ev- In this eMag we provide both implementation ery component. The number of jobs in the contin- examples and comparisons of different possible ap- uous integration/delivery server explodes and a proaches on a range of topics from immutable infra- maintenance nightmare ensues if care is not taken. structure to self-service ops platforms and service Automated job creation and update becomes cru- discovery. In addition, we talk about the Docker eco- cial to keep pace and avoid development teams get- system and the different aspects to consider when ting bogged down by build and deployment issues. moving to this increasingly popular system for ship- Martin Peston provides an overview and practical ping and running applications. examples on this topic. In the immutable infrastructure virtual panel we When adopting immutable infrastructure the learn from both advocates and skeptics on the pros need to (re)connect services dynamically becomes and cons of this “disposable” approach to infrastruc- a priority. This process is called Service Discovery. ture changes. Instead of trying to converge machines Microservice architectures further exacerbate the to a desired state, outdated machine are simply de- importance of reliably discovering and connecting stroyed and replaced by a brand new machine with an increasing number of standalone services. The the new changes applied. Simplicity Itself team analyzed the main solutions Chris Swan gives us a rundown of the increas- available, as well as the pros and cons of (trying to) ingly popular Docker tool, outlining the strengths do this on your own.

4 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 READ ONLINE ON InfoQ

Virtual Panel on Immutable Infrastructure

By Manuel Pais

THE PANELISTS

Chad Fowler is an internationally known software developer, trainer, manager, speaker, and musician. Over the past decade, he has worked with some of the world’s largest companies and most admired software developers. Chad is CTO of 6 Wunderkinder. He is the author or co-author of a number of popular software books, including Rails Recipes and The Passionate Programmer: Creating a Remarkable Career in .

Mark Burgess is the CTO and founder of CFEngine, formerly professor of network and system administration at Oslo University College, and the principal author of the CFEngine software. He’s the author of numerous books and papers on topics from physics to network and system administration to fiction.

Mitchell Hashimoto is best known as the creator of Vagrant and founder of HashiCorp. He is also an O’Reilly author and professional speaker. He is one of the top GitHub users by followers, activity, and contributions. “Automation obsessed”, Mitchell strives to build elegant, powerful DevOps tools at HashiCorp that automate anything and everything. Mitchell is probably the only person in the world with deep knowledge of most virtualization hypervisors.

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 5 Servers accumulate change for better or worse over time, including: new applications, upgrades, configuration changes, scheduled tasks, and in the moment fixes for issues. One thing is certain, the longer a server has been provisioned and running the more likely it is in an unknown state. Immutable Servers solve the problem of being certain about server state by creating them anew for each change described previously. Immutable servers make infrastructure scalable and reliable in spite of change. However, this can require fundamentally new views of systems, patterns, deployments, application code and team structure, says panel member Chad Fowler.

Mitchell Hashimoto, another in functional programming have. one: it would be frozen and noth- panelist, sees immutable infra- Infrastructure components, like ing would happen). What the structure as an enabler for re- in-memory data structures, are term tries to capture is the idea peatable environments (through running components that can that one should try to predeter- configuration management be accessed concurrently. So the mine as many of the details as tools) and extremely fast deploy- same problems of shared state possible of a server configuration ments times (through pre-built exist. at the disk-image stage, so that static images) and service or- In my definition of im- «no configuration is needed». chestration times (through de- mutable infrastructure, servers Then the idea supposes that this centralized orchestration tools). (or whatever) are deployed once will make it fast and reliable to This idea has been deemed and not changed. If they are spin up machines. When some- utopian by Mark Burgess, our changed for some reason, they thing goes wrong, you just dis- third panelist, due to the large are marked for garbage collec- pose of the machine and rebuild number of external dependen- tion. Software is never upgraded a new one from scratch. That is cies in any system today. Burgess on an existing server. Instead, my understanding of what peo- criticizes the “abuse of the notion the server is replaced with a new, ple are trying to say with this of immutability”. functionally equivalent server. phrase. InfoQ reached out to them Mitchell: Immutable infrastruc- I believe there are a number to ask about their definition and ture is treating the various com- of things wrong with this argu- borders of immutable infrastruc- ponents of your infrastructure ment though. The idea of what it ture as well as its benefits and as immutable, or unchangeable. means to fix the configuration is drawbacks, in particular when Rather than changing any com- left very unclear and this makes compared to “desired state” con- ponent, you create a new one the proposal unnecessarily con- figuration management solu- with the changes and remove tentious. First of all, predeter- tions. Is it a step forward or back- the old one. As such, you can be mining everything about a host wards in effective infrastructure certain (or more confident) that is not possible. Proper configura- management? if your infrastructure is already tion management deals with dy- functioning, a change to config namic as well as static host state. management, for example, won’t Take the IP address and network- break what’s already not broken. ing config, for instance – this InfoQ: Can you briefly ex- Immutable infrastructure has to be set after the machine plain your definition of im- is also sometimes referred to as is spun up. What about execut- mutable infrastructure? “phoenix servers” but I find that ing new programs on demand? term to be less general, since im- What if programs crash, or fail to mutability can also apply at the start? Where do we draw the line Chad: I wrote about this on service level, rather than just the between what can and can’t be my blog several months ago in server level. done after the machine has been detail, but to me the high-level Mark: The term «immutable» is spun up? Can we add a package? essence of immutable infrastruc- really a misnomer (if infrastruc- Should we allow changes from ture shares the same qualities ture were really immutable, it DHCP but not from CFEngine or that immutable data structures would not be much use to any- Puppet? Why? In my view, this is

6 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 all change. So no machine can be afraid of destroying a com- be immutable in any meaningful ponent, so when a new version InfoQ: Is immutable infra- sense of the word. is available, you should be able structure a better way to The real issue we should to eventually replace every com- handle infrastructure than focus on is what behaviours we ponent. Therefore, your infra- desired-state convergence? want hosts to exhibit on a con- structure is highly convergent. If yes, what are its main ad- tinuous basis. Or, in my language, However, this is mostly a disci- vantages? If not, what are its what promises should the infra- pline-and-process thing, which is main disadvantages? structure be able to keep? sometimes difficult to enforce in an organization. Chad: I think so. Of course, Mark: I don’t believe immutable there are tradeoffs involved and infrastructure helps prevent sys- you have to weigh the options InfoQ: Can immutable infra- tems divergence. Trying to freeze in every scenario, but I think structure help prevent sys- configuration up front leads to immutable infrastructure is a tems divergence while still a “microwave dinner” mental- better default answer than de- coping with the need for reg- ity. Just throw your prebaked sired-state convergence. ular configuration updates? package in the oven and suffer Immutable servers are easi- through it. It might be okay for er to reason about. They hold up Chad: Absolutely. It’s just a dif- some people, but then you have better in the face of concurrency. ferent model for updates. Rather two problems: either you can’t They are easier to audit. They are than update an existing system, get exactly what you need or you easier to reproduce, since the ini- you replace it. Ultimately, I think have a new problem of making tial state is maintained. this is a question of granularity and managing sufficient variety Mitchell: It has its benefits and of what you call a “component”. of prepackaged stuff. The latter it has its downsides. Overall, I Fifteen years ago, if I wanted to is a harder problem to solve than believe it to be a stronger choice update a software component just using fast model-based con- and the right way forward, but it on a UNIX system, I upgraded the figuration management because is important to understand it is software package and its depen- it’s much harder to see into pre- no silver bullet, and it will intro- dencies. Now I tend to view run- packaged images or “microwave duce problems you didn’t have ning server instances as compo- dinners” to see what’s in them. So before (while fixing others). nents. If you need to upgrade the you’d better get your packaging The advantages are deploy- OS or some package on the sys- exactly right. ment speed, running stability, tem, just replace the server with Moreover, what happens development testability, version- one that’s updated. If you need if there is something slightly ing, and the ability to roll back. to upgrade your own application wrong? Do you really want to go With immutable, because code, create new server instanc- back to the factory and repack- everything is “precompiled” into es with the new code and replace age everything just to follow the an image, deployment is ex- the old servers with it. dream of the microwave meal? tremely fast. You launch a server, Mitchell: Actually, immutable It is false that prepackaging is and it is running. There may be infrastructure makes things a bit the only way to achieve consis- some basic configuration that worse for divergence if you don’t tency. Configuration tools have happens afterwards but the slow practice it properly. With mu- proven that. You don’t need to parts are done: compiling soft- table infrastructure, the idea is destroy the entire machine to ware, installing packages, etc. that configuration management make small repeatable changes And because everything is constantly runs (on some inter- cheaply. Would you buy a new immutable, once something is val) to keep the system in a con- car because you have a flat tyre, running, you can be confident vergent state. With immutable or because it runs out of fuel? that an external force won’t be as infrastructure, you run the risk of What prevents divergence likely to affect stability. For exam- deploying different versions of of systems is having a clear mod- ple, a broken configuration-man- immutable pieces, resulting in a el of the outcome you intend agement run cannot accidentally highly divergent environment. – not the way you package the corrupt configuration. This mistake, however, is starting point from which you Immutable infrastructure is the result of not properly em- diverge. incredibly easy to test, and the bracing or adopting immutable test results very accurately show infrastructure. With immutable what will actually happen at run infrastructure, you should never time. An analogy I like to make

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 7 is that immutable infrastructure the answer unless every transac- waste returns. At that stage, we is to configuration management tion is designed to be carried out start to discover that the dispos- what a compiled application is in separately built infrastructure. able scheme was actually toxic to to source code. You can unit-test That’s a political oversimplifica- the environment (what are the your source code, but when you tion, not a technical one, and it side effects of all this waste?), go to compile it, there is no guar- adds overhead. and we wonder why we were not antee that some library versions I would say that the “im- thinking ahead. We are current- that could ruin your build didn’t mutable” paradigm is generally ly in an age of apparent plenty, change. Likewise, with configura- worse than one that that bal- with cloud environments hiding tion management, you can run it ances a planned start image the costs of waste, so that devel- over and over, but you can’t guar- with documented convergent opers don’t have to think about antee that if it succeeds that it’ll adaptation. The disadvantage of them. But I wonder when the still succeed months down the a fixed image is a lack of trans- margins will shrink to the point road. But with a compiled appli- parency and immediacy. Advo- where we change our minds. cation, or a prebuilt server, all the cates of it would probably argue dependencies are already sat- that if they know what the disk isfied and baked in; the surface image version is, they have a bet- area of problems that can hap- ter idea of what the state of the InfoQ: What are the borders pen when you go to launch that machine is. The trouble with that for immutable infrastructure, server are much, much smaller. is that system state is not just i.e., for which kind of chang- Versioning is much simpler what you feed in at the start – it es would it make sense to and clearer because you can tag also depends on everything that replace a server vs. updating a specific image with the config- happens to it after it is running. its configuration/state? uration-management revision It promotes a naive view of state. that is baked into it, the revision At a certain scale, pre-cach- Chad: If there are borders and of an application, the versions of ing some decision logic as a fixed some changes are done on an all dependencies, etc. With mu- image might save you a few sec- existing server, the server isn’t table servers, it’s harder to be onds in deployment, but you immutable. Idealistically, I don’t certain what versions or revisions could easily lose those seconds think we should allow borders. of what exist on each server. (and a lot more business conti- “Immutable” isn’t a buzzword. It Finally, you get rollback ca- nuity) by having to redeploy ma- has meaning. We should either pability! There are many people chines instead of adapting and maintain the meaning or stop who think rollback is a lie, and at repairing simple issues. If there using the word. An in-between some point it is. But if you prac- is something wrong with your is dangerous and may provide a tice gradual, incremental chang- car, it gets recalled for a patch; false sense of security in the per- es to your infrastructure, rolling you don’t get a new car, else the ceived benefits of immutability. back with immutable infrastruc- manufacturers would be out of That said, systems do need ture to a recently previous ver- business. to be quickly fixable, and the sion is cheap and easy: you just Caching data can certain- methods we’re currently using replace the new servers with ly make sense for optimizing for replacing infrastructure are servers launched from a previous effort, as part of an economy of slower than hot-fixing an exist- image. This has definitely saved scale, but we should not turn this ing server. So there needs to be us some serious problems a few into a false dichotomy by claim- some kind of acceptable hybrid times, and is very hard to achieve ing it is the only way. In general, which maintains the benefits of with desired-state configura- an approach based on partial immutability. My current plan tions. disk images, with configuration for Wunderlist is to implement a Mark: The way to get infrastruc- management for the “last mile” hotfix with a self-destruct built ture right is to have a continuous changes makes much more busi- in. So if you have to hot-fix a serv- knowledge relationship with ness sense. er, it gets marked to be replaced system specifications (what I call In practice, “immutability” automatically. We haven’t done promises in CFEngine language). (again poor terminology) means it automatically yet, but we’ve To be fit for purpose, and to sup- disposability. Disposability manually done this and it works port business continuity, you emerges often in a society when well. I see this as an ugly optimi- must know that you can deliver resources seem plentiful. Eventu- zation rather than a good design continuous consistency. Chang- ally resources become less plen- approach. ing the starting point cannot be tiful, and the need to handle the

8 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 Mitchell: The borders of im- parallel as separate branches for the server, you might be destroy- mutable infrastructure for me some kinds of applications. But ing the state as well, which is break down to where you want we have to remember that the usually unacceptable. to be able to change things rap- whole world is not in the cloud. Mark: Mission-critical servers idly: small configuration chang- Planes, ships, rockets, mobile running monolithic applications es, application deploys, etc. devices are all fragile to change cannot generally be managed But wanting to be able to and mission critical. There are in this disposable manner. As I deploy an application on an im- thus still embedded devices that understand it, the principal ar- mutable server doesn’t make spend much of their time large- gument for this pattern of work- that server immutable. Instead, ly offline, or with low-rate com- ing is one of trust. Some people you should think of the immuta- munications. They cannot be would rather trust an image than bility of a server like an onion: it re-imaged and replaced safely a configuration engine. One has layers. The base layer of the or conveniently, but they can be would like to allow developers to server (the OS and some con- patched and managed by some- manage and maintain their own figuration and packages) is im- thing like CFEngine that doesn’t infrastructure requirements in- mutable. The application itself is even need a network connection creasingly. However, if you force its own immutable component: to function. everyone to make changes only a hopefully precompiled binary through disk images, you are ty- being deployed to the server. So ing their hands with regard to while you do perhaps have an making dynamic changes, such arguably mutable component in InfoQ: Is it possible to treat as adjusting the number of paral- your server, it itself is another ver- every server of a given in- lel instances of a server to handle sioned immutable component. frastructure as immutable or latency and tuning other aspects What you don’t want to be are there some server roles of performance. Disregarding doing for application deploys on that cannot be treated that those concerns is a business de- immutable infrastructure is to be way? cision. Developers often don’t compiling live on an immutable have the right experience to un- server. The compilation might Chad: We have what I call derstand scalability and perfor- fail, breaking your application «cheats» with immutable infra- mance, and certainly not in ad- and perhaps the functionality of structure. Relational databases vance of deployment. the server. are a good example. I think it’s Mark: When a particular con- possible to work with them im- figuration reaches the end of mutably, but so far it hasn’t been its useful life, it should probably worth the effort for us. If we were InfoQ: What are the main con- be replaced. That ought to be a an infrastructure vendor, I would ceptual differences between business judgment, not a tech- be applying some effort here, tooling for immutable infra- nical one. The judgment can be but since we’re in the business of structure and desired-state made on economic grounds, re- making applications for our cus- convergence? lated to what would be lost and tomers, we have been content gained by making a change. But to outsource more and more to Chad: Desired-state conver- be careful of the hidden costs if managed services such as Ama- gence is in a different realm of your processes are not transpar- zon RDS. complexity to implement. It’s a ent and your applications are My goal is that our entire fascinating idea, but at least for mission critical. infrastructure consists of either my use cases it’s outdated. The Anytime you have to bring pieces we don’t manage direct- longer a system lives, the more down a system for some reason, ly or components that are com- afraid of it I become. I can’t be it could be an opportunity to pletely replaceable. We’re almost 100% sure that it is configured replace it entirely without un- there and so far it’s been a very the way I want and that it has necessary interruption, as long positive experience. exactly the right software. Thou- as you have a sufficient model Mitchell: It is possible, but there sands upon thousands of devel- of the requirements to replace are roles that are much easier oper hours have gone into solv- it with minimum impact, and a to treat as immutable. Stateless ing this problem. hands-free approach to automa- servers are extremely easy to In the world of immutable tion for creating that environ- make immutable. Stateful serv- infrastructure, I think of servers ment. Today, it is getting easy ers such as databases are much as replaceable building blocks, to deploy multiple versions in trickier, because if you destroy like parts in an automobile. You

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 9 don’t update a part. You just re- pairs and even simple changes imenting with and testing for place it. The system is the sum of in real time and respond to busi- some aspects of your infrastruc- its parts. The parts are predict- ness problems on the fly, at pret- ture, though. able since they don’t change. It’s ty much any scale, making only Mark: I don’t believe it can be- conceptually very simple. minimal interventions. This is like come mainstream unless all Mitchell: The tooling doesn’t the role of cellular DNA in biolo- software becomes written in a change much! Actually, all tools gy. There are repair processes on- completely stateless way, which used for desired-state conver- going because there is no redun- would then be fragile to com- gence are equally useful for im- dancy at the intracellular level. munication faults in a distributed mutable infrastructure. Instead, Bulk information is easier world. Even then, I don’t think it immutable infrastructure adds to manage from a desired-state is desirable. Do we really want a compilation step to servers model than from piles of bulk to argue that it is better for the or applications that didn’t exist data because it exploits patterns whole world to eat microwave before. For example, instead of to good advantage. You can eas- dinners, or to force chefs to pack- launching a server and running ily track changes to the state (for age things in plastic before eat- , you now use Packer as a compliance and auditing purpos- ing it? If history has taught us compilation tool to launch a es) because a model defines your anything, it is that people crave server, run Chef, and turn it into standard of measurement over freedom. We have to understand an image. many versions. Imagine compli- that disposability is a large-scale One thing that does change ance auditing like PCI or HIPPA. economic strategy that is just not is a mindset difference: im- How do you prove to an auditor suitable at all scales. mutable tools know they can that your system is compliant? If only run once to build an image you don’t have a model with de- whereas desired-state conver- sired outcome, that becomes a gence tools expect that they can process of digging around in files InfoQ: Is immutable infra- run multiple times to achieve and looking for version strings. structure applicable for or- convergence. In practice, this It’s very costly and time wasting. ganizations running their doesn’t cause many problems services on physical infra- because you can just run the structure (typically for per- desired-state convergence tool formance reasons)? If so, multiple times when building an InfoQ: Is the choice and ma- how? image. However, the tools built turity of today’s tools and for immutability tend to be much the immutable-infrastructure Chad: Sure. While perhaps the more reliable in achieving their pattern itself enough to be- servers themselves would run intended purpose the first time. come mainstream? longer and probably require in- Mark: If you want to freeze the place upgrades to run efficiently, configuration of a system to a Chad: Probably not. The founda- with the many options available predefined image, you have to tions are getting better and bet- for virtualization, everything on have all the relevant information ter with both hosted and internal top of that is fair game. I sup- about its environment up front, cloud providers and frameworks, pose it would be possible to take and then you are saying that but creating a solid immutable the approach further down the you won’t try to adapt down the architecture is not currently the stack, but I haven’t had to do it line. You will kill a host to repair path of least resistance. I think and I don’t want to speculate. the smallest headache. It’s an most of us will move in this direc- Mitchell: Yes, but it does require overtly discontinuous approach tion over time, but it’s currently a bit more disciplinary work. The to change, as opposed to one far from mainstream. organization needs to have in based on preservation and con- Mitchell: Not yet, but they’re place some sort of well automat- tinuity. If you think of biology, it’s leaps and bounds better than ed process for disposing of and like tissue, where you can lose a they were a year ago, and they’re re-imaging physical machines. few cells from your skin because only going to continue to be- Unfortunately, many organiza- there are plenty more to do the come more mature and solve the tions do not have this, which is job. It can only work if resources various problems early adopters somewhat of a prerequisite to are plentiful and redundant. of immutable infrastructure may making immutable infrastruc- With desired-state conver- be having. ture very useful. gence, you can make a system All the tools are definitely completely predictable with re- mature enough to begin exper-

10 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 For example, what you real- Developers in an immutable infra- ly want is something like Mesos or structure have to constantly keep One thing is Borg for physical hardware. in mind that any service they talk to Mark: The immutable-infrastruc- can die at any moment (and hope- ture idea is not tied specifically to fully can be replaced rather quick- certain, the longer virtualization. The same method ly). This mindset alone results in de- could be applied to physical infra- velopers generally building much a server has structure, but the level of service more robust and failure-friendly discontinuity would be larger. applications. Today, immutability is often With strongly coupled, mu- been provisioned being mixed up with arguments for table infrastructures, it isn’t uncom- continuous delivery in the cloud, mon to interrupt a dependent ser- and running the but I believe that disposable com- vice of an application and have that puting could easily be contrary to application be completely broken the goals of continuous delivery until it is restarted with the depen- more likely it is because it adds additional hoops dent service up. to jump through to deploy change, While keeping immutable in- in an unknown and makes the result less transpar- frastructure in mind, applications ent. are much more resilient. As an ex- state. Immutable ample from our own infrastructure managing Vagrant Cloud, we were able to replace and upgrade every Servers solve the InfoQ: With respect to an ap- server (our entire infrastructure) plication’s architecture and without any perceivable downtime problem of being implementation, what factors and without touching the web front need to be taken into account ends during the replacement pro- when targeting an immutable cess. The web applications just re- certain about infrastructure? tried some connects over time and eventually came back online. The server state by Chad: Infrastructure and services only negative experience was that need to be discoverable. Whatev- for some people their requests were er you’re using to register services queued a bit longer than usual! creating them needs to be programmable via an Mark: The aim should not be to API. You need to have intelligent make applications work around an anew for each monitoring and measuring in place, immutable infrastructure. You don’t and your monitoring needs to be pick the job to fit the tools. The focused less on the raw infrastruc- immutable infrastructure is usual- change described ture than on the end purpose of the ly motivated as a way of working service than you’re probably used around the needs of application de- previously to. velopers. The key question is how Everything needs to be state- do you optimize a continuous-de- less where possible. livery pipeline? As I mentioned previously, we have cheats like managed database systems. I’m not going to com- ment on how you have to change InfoQ: Does immutable infra- architecture to have your own structure promote weak root- immutable, disposable database cause analysis as it becomes systems since it’s thankfully not a so easy to trash a broken serv- problem I’ve needed or wanted to er to repair service instead of solve yet. fixing it? If so, is that a problem Mitchell: Nothing has to change or just a new modus operandi? in order to target immutable in- frastructure, but some parts of Chad: I don’t think so. In the worst architecting and developing an case, it doesn’t change how we do application become much easier. root-cause analysis, because re-

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 11 placing and rebooting are sort of ble scripts and pre-templated the same last-ditch effort when files, where the reason for the de- Immutable servers something is wrong. In the best cisions only lives in some devel- case, it makes it easier to exper- oper’s head. That process might are easier to iment with possible solutions, work to some extent if the devel- tweak variables one at a time, oper is the only one responsible bring up temporary test servers, for making it, but it makes repro- reason about. They swap production servers in and ducibility very challenging. What out, etc. happens when the person with hold up better I see the point you’re hint- that knowledge leaves the orga- ing at in the question, though. nization? There may be a class of problems In psychology, one knows in the face of that is all but eliminated (read: that humans cannot remember obscured to the point of essen- more than a small number of concurrency. They tially not existing) if the average things without assistance. The lifespan of a server is less than question is how do you create one day. It may also be harder to a knowledge-oriented frame- are easier to audit. pinpoint these problems if they work where intent and outcome do occasionally pop up without are transparent and quickly re- They are easier to knowing to try server longevity producible with a minimum as a variable. of repeated effort. This is what Maybe that’s okay. configuration management was reproduce, since Mitchell: Because it is very likely designed for and I still believe that the server you’re replacing it that it is the best approach to the initial state is with will one day see that same managing large parts of the in- issue, immutable infrastructure frastructure configuration. The maintained. doesn’t promote any weaker key to managing infrastructure root-cause analysis. It may be is in separating and adapting easier to ignore for a longer pe- different behaviours at relevant riod of time, but most engineer- scales in time and space. I have ing organizations will care to fix written a lot about this in my it properly at some point. book In Search of Certainty: The Actually, I would say the Science of Our Information Infra- root cause analysis becomes structure. much stronger. Since the compo- nent is immutable and likely to exhibit the same problems under the same conditions, it is easier to reproduce, identify, fix, and finally to deploy your change across your entire infrastructure. Additionally, desired-state configuration has a high chance of making the problem worse: a scheduled run of the configura- tion management system may mask the real underlying issue, causing the ops team to spend more time trying to find it or even to just detect it. Mark: Disposal of causal evi- dence potentially makes un- derstanding the environment harder. Without model-based configuration, a lot of decisions get pushed down into inscruta-

12 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 READ ONLINE ON InfoQ

Docker: Present and Future

Chris Swan is CTO at CohesiveFT, a provider of cloud networking software. Previously, he spent a dozen years in financial services as a technologist to bankers and banker to technologists. He spent most of that time at large Swiss banks wrangling infrastructure for app servers, compute grids, security, mobile, and cloud. Chris also enjoys tinkering with the Internet of Things, including a number of Raspberry Pi projects.

Docker is a toolset for Linux containers designed to build, ship, and run distributed applications, first released as an open-source project by dotCloud in March 2013. The project quickly became popular, leading dotCloud to rebrand as Docker, the company (and ultimately selling off their original PaaS business). The company released Docker 1.0 in June 2014 and has since sustained the monthly release cadence that led up to the June release.

The 1.0 release marked the point sion of Docker can be used with zon, Google, and Microsoft. It is where the company considered any other version (with both for- almost ubiquitously available the platform sufficiently mature ward and backward compatibil- wherever Linux can be found. In for use in production (with the ity), something that provides a addition to the big names, many company and partners selling stable foundation for Docker use startups are growing up around support options). The monthly despite rapid change. Docker or changing direction to release of point updates shows The growth of Docker into better align with Docker. Those that the project is still evolving one of the most popular open- partnerships (large and small) quickly, adding new features, source projects could be per- are helping to drive rapid evo- and addressing issues as they are ceived as hype, but there is a lution of the core project and its found. The project has success- great deal of substance. Dock- surrounding ecosystem. fully decoupled “ship” from “run”, er has attracted support from A brief technical overview so images sourced from any ver- the industry, including Ama- of Docker

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 13 Docker makes use of Li- they’re able to share a single and public clouds. It’s also pos- nux-kernel facilities such kernel and application librar- sible to run VMs inside contain- as cgroups, namespaces, and ies. This can lead to substantial- ers, which is something that SELinux to provide isolation be- ly smaller RAM footprints even Google uses as part of its cloud tween containers. At first, Docker when compared to virtualisation platform. Given the widespread was a front end for the LXC con- systems that can make use of availability of infrastructure as a tainer-management subsys- RAM over-commitment. Storage service (IaaS) that provides VMs tem, but release 0.9 introduced footprints can also be reduced on demand, it’s reasonable to libcontainer, which is a native when deployed containers expect that containers and VMs Go-language library that pro- share underlying image layers. will be used together for years vides the interface between user IBM’s Boden Russel has bench- to come. It’s also possible that space and the kernel. marked these differences. container management and vir- Containers sit on top of a Containers also present a tualisation technologies might union file system, such as AUFS, lower systems overhead than be brought together to provide which allows for the sharing of VMs, so the performance of an a best-of-both-worlds approach; components such as operat- application inside a container so a hardware trust-anchored ing-system images and installed will generally be the same or micro-virtualisation implemen- libraries across multiple contain- better than the same application tation behind libcontainer could ers. The layering approach in running within a VM. A team of integrate with the Docker tool the file system is also exploited IBM researchers have published chain and ecosystem at the front by the Dockerfile DevOps tool, a performance comparison of end, but use a different back which is able to cache opera- virtual machines and Linux con- end that provides better isola- tions that have already success- tainers. tion. Micro virtualisation (such fully completed. This can greatly One area where containers as Bromium vSentry and VM- speed up test cycles by eliminat- are weaker than VMs is isolation. ware’s Project Fargo) is already ing the wait time usually taken to VMs can take advantage of ring used in desktop environments install operating systems and ap- -1 hardware isolation such as that to provide hardware-based iso- plication dependencies. Shared provided by Intel’s VT-d and VT-x lation between applications, libraries between containers can technologies. Such isolation pre- so similar approaches could be also reduce RAM footprint. vents VMs from breaking out and used along with libcontainer as A container is started from interfering with each other. Con- an alternative to the container an image, which may be locally tainers don’t yet have any form mechanisms in the Linux kernel. created, cached locally, or down- of hardware isolation, which loaded from a registry. Docker, makes them susceptible to ex- “Dockerizing” Inc. operates the Docker Hub ploits. A proof-of-concept attack applications Registry, which hosts official re- named Shocker showed that Pretty much any Linux applica- positories for a variety of oper- Docker versions prior to 1.0 were tion can run inside a Docker con- ating systems, middleware, and vulnerable. Although Docker 1.0 tainer. There are no limitations databases. Organisations and fixed the particular issue exploit- on choice of languages or frame- individuals can host public re- ed by Shocker, Docker CTO Solo- works. The only practical limita- positories for images at Docker mon Hykes said, “When we feel tion is what a container is allowed Hub, and there are also subscrip- comfortable saying that Docker to do from an operating-system tion services for hosting private out of the box can safely contain perspective. Even that bar can be repositories. Since an uploaded untrusted uid0 programs, we will lowered by running containers image could contain almost any- say so clearly.” Hykes’s statement in privileged mode, which sub- thing, the Docker Hub Registry acknowledges that other exploits stantially reduces controls (and provides an automated build fa- and associated risks remain, and correspondingly increases risk of cility (previously called “trusted that more work will need to be the containerised application be- build”) where images are con- done before containers can be- ing able to cause damage to the structed from a Dockerfile that come trustworthy. host ). serves as a manifest for the con- For many use cases, the Containers are started from tents of the image. choice of containers or VMs is a images, and images can be made false dichotomy. Docker works from running containers. There Containers versus VMs well within a VM, which allows are essentially two ways to get Containers are potentially much it to be used on existing virtu- applications into containers: more efficient than VMs because al infrastructure, private clouds, manually and Dockerfile.

14 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 Manual builds container, a supervisor process Connectivity A manual build starts by launch- must be launched that can then Docker’s networking capabili- ing a container with a base oper- spawn the other desired process- ties are fairly primitive. Services ating-system image. An interac- es. There is no init system within within containers can be made tive terminal can then be used to containers, so anything that re- accessible to other containers on install applications and depen- lies on systemd, upstart, or simi- the same host, and Docker can dencies using the package man- lar won’t work without modifica- also map ports onto the host op- ager offered by the chosen fla- tion. erating system to make services vour of Linux. Zef Hemel provides available across a network. The a walk through of the process in Containers and officially sponsored approach his InfoQ article “Docker: Using microservices to connectivity is libchan, which Linux Containers to Support Por- A full description of the phi- is a library that provides Go- table Application Deployment”. losophy and benefits of using like channels over the network. Once the application is installed, a microservices architecture is Until libchan finds its way into the container can be pushed to a beyond the scope of this article applications, there’s room for registry (such as Docker Hub) or (and is well covered in the InfoQ third parties to provide comple- exported into a tar file. e-mag Microservices). Containers mentary network services. For are however a convenient way to example, Flocker has taken a Dockerfile bundle and deploy instances of proxy-based approach to make Dockerfile is a system for script- microservices. services portable across hosts ing the construction of Docker Whilst most practical exam- (along with their underlying stor- containers. Each Dockerfile spec- ples of large-scale microservices age). ifies the base image to start from deployments to date have been and then a series of commands on top of (large numbers of) VMs, Compositing that are run in the container and/ containers offer the opportunity Docker has native mechanisms or files that are added to the con- to deploy at a smaller scale. The for linking containers together tainer. The Dockerfile can also ability for containers to have a by which metadata about a de- specify ports to be exposed, the shared RAM and disk footprint pendency can be passed into working directory when a con- for operating systems and librar- the dependent container and tainer is started, and the default ies and common application consumed within as environ- command on start up. Contain- code also means that deploying ment variables and hosts en- ers built with Dockerfiles can multiple versions of services side tries. Application compositing be pushed or exported just like by side can be made very effi- tools like Fig and geard express manual builds. Dockerfiles can cient. the dependency graph inside a also be used in Docker Hub’s single file so that multiple con- automated build system so that Connecting containers tainers can be brought together images are built from scratch in Small applications will fit inside a into a coherent system. Centu- a system under the control of single container, but in many cas- ryLink’s Panamax compositing Docker, Inc. with the source of es an application will be spread tool takes a similar underlying that image visible to anybody across multiple containers. Dock- approach to Fig and geard, but that might use it. er’s success has spawned a flurry adds a web-based user inter- of new application compositing face, and integrates directly with One process? tools, orchestration tools, and GitHub so that applications can Whether images are built man- platform as a service (PaaS) im- be shared. ually or with Dockerfile, a key plementations. Behind most of consideration is that only a sin- these efforts is a desire to sim- Orchestration gle process is invoked when the plify the process of constructing Orchestration systems like deck- container is launched. For a con- an application from a set of in- ing, New Relic’s Centurion and tainer serving a single purpose, terconnected containers. Many Google’s Kubernetes all aim to such as running an application tools also help with scaling, fault help with the deployment and server, running a single process tolerance, performance manage- lifecycle management of con- isn’t an issue (and some argue ment, and of de- tainers. There are also numerous that containers should only have ployed assets. examples (such as Mesosphere) a single process). For situations of Apache Mesos (and partic- where it’s desirable to have mul- ularly its Marathon framework tiple processes running inside a for long-running applications)

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 15 being used along with Dock- date kernel, it can run in pretty Docker and the distros er. By providing an abstraction much every cloud that offers Docker has already become a between the application needs IaaS. Many of the major cloud standard feature of major Linux (e.g. expressed as a requirement providers have announced addi- distributions like Ubuntu, Red for CPU cores and memory) and tional support for Docker and its Hat Enterprise Linux (RHEL), and underlying infrastructure, the or- ecosystem. CentOS. Unfortunately, the distri- chestration tools provide decou- Amazon has introduced butions move at a different pace pling that’s designed to simplify Docker in its Elastic Beanstalk to the Docker project, so the ver- both application development system (which is an orchestration sions found in a distribution can and datacentre operations. There service over underlying IaaS). be well behind the latest avail- is such a variety of orchestration Google has Docker-enabled able. For example, Ubuntu 14.04 systems because many have managed VMs, which provide was released with Docker 0.9.1, emerged from internal systems a halfway house between the and that didn’t change on the previously developed to manage PaaS of App Engine and the IaaS point release upgrade to Ubun- large-scale deployments of con- of Compute Engine. Microsoft tu 14.04.1 (by which time Dock- tainers; for example Kubernetes and IBM have both announced er was at 1.1.2). There are also is based on Google’s Omega sys- services based on Kubernetes so namespace issues in official re- tem that’s used to manage con- that multi-container applications positories since Docker was also tainers across the Google estate. can be deployed and managed the name of a KDE system tray Whilst there is some degree on their clouds. – so with Ubuntu 14.04 the pack- of functional overlap between To provide a consistent in- age name and command-line the compositing tools and the terface to the wide variety of back tool are both “docker.io”. orchestration tools there are also ends now available, the Dock- Things aren’t much differ- ways that they can complement er team introduced libswarm ent in the enterprise Linux world. each other. For example, Fig (now Docker Swarm) to inte- CentOS 7 comes with Docker might be used to describe how grate with a multitude of clouds 0.11.1, a development release containers interact functionally and resource-management that precedes Docker, Inc.’s an- whilst Kubernetes pods might be systems. One of the stated aims nouncement of production used to provide monitoring and of libswarm is to “avoid vendor readiness with Docker 1.0. Linux scaling. lock-in by swapping any ser- distribution users that want the vice out with another”. This is latest version for promised sta- Platforms (as a service) accomplished by presenting a bility, performance, and security A number of Docker native consistent set of services (with would be better off following PaaS implementations such associated APIs) that attach to the installation instructions and as Deis and Flynn have emerged implementation-specific back using repositories hosted by to take advantage of the fact that ends. For example, the Docker Docker, Inc. rather than taking Linux containers provide a great server service presents the Dock- the version included in their dis- degree of developer flexibility er remote API to a local Docker tribution. (rather than being “opinionated” command-line tool so that con- The arrival of Docker has about a given set of languages tainers can be managed on an spawned new Linux distributions and frameworks). Other plat- array of service providers. such as CoreOS and Red Hat’s forms such as Cloud Foundry, New service types based on Project Atomic that are designed OpenShift, and Apcera’s Continu- Docker are still in their infancy. to be a minimal environment um have taken the route of inte- London-based Orchard offered for running containers. These grating Docker-based function- a Docker hosting service, but distributions come with new- ality into their existing systems, Docker, Inc. said that the service er kernels and Docker versions so that applications based on wouldn’t be a priority after ac- than the traditional distributions. Docker images (or the Docker- quiring Orchard. Docker, Inc. has They also have lower memory files that make them) can be de- also sold its previous dotCloud and disk footprints. The new dis- ployed and managed alongside PaaS business to cloudControl. tributions also come with new apps using previously supported Services based on older contain- tools for managing large-scale languages and frameworks. er-management systems such deployments such as fleet, “a dis- as OpenVZ are already com- tributed init system” and etcd for All the clouds monplace, so to a certain extent metadata management. There Since Docker can run in any Li- Docker needs to prove its worth are also new mechanisms for nux VM with a reasonably up-to- to hosting providers. updating the distribution itself

16 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 so that the latest versions of the The future of Docker kernel and Docker can be used. Docker, Inc. has set a clear path For many use This acknowledges that one of to the development of core capa- the effects of using Docker is bilities (libcontainer), cross-ser- that it pushes attention away vice management (libswarm), cases the choice from the distribution and its and messaging between con- package-management solution, tainers (libchan). Meanwhile, the of containers or making the Linux kernel (and the company has already shown a Docker subsystem using it) more willingness to consume its own VMs is a false important. ecosystem with the Orchard ac- New distributions might be quisition. There is however more the best way to run Docker, but to Docker than Docker, Inc., with dichotomy. traditional distributions and their contributions to the project com- package managers remain very ing from big names like Google, Docker works important within containers. IBM, and Red Hat. With a benev- Docker Hub hosts official images olent dictator in the shape of for Debian, Ubuntu, and CentOS. CTO Solomon Hykes at the helm, well within a VM, There’s also a semi-official repos- there is a clear nexus of technical itory for Fedora images. RHEL leadership for both the company which allows it images aren’t available in Docker and the project. Over its first 18 Hub, as they’re distributed direct- months, the project has shown ly from Red Hat. This means that an ability to move fast by using to be used on the automated build mechanism its own output, and there are no on Docker Hub is only available signs of that abating. existing virtual to those using pure open-source Many investors are look- distributions (and willing to trust ing at the features matrix for the provenance of the base imag- VMware’s ESX/vSphere platform infrastructure, es curated by the Docker team). from a decade ago and figuring Whilst Docker Hub inte- out where the gaps (and oppor- private clouds and grates with source-control sys- tunities) lie between enterprise tems such as GitHub and Bit- expectations driven by the pop- public clouds. bucket for automated builds, ularity of VMs and the existing the package managers used Docker ecosystem. Areas like during the build process create a networking, storage, and fine- complex relationship between a grained version management build specification (in a Docker- (for the contents of containers) file) and the image resulting from are presently underserved by the a build. Non-deterministic results existing Docker ecosystem, and from the build process aren’t spe- provide opportunities for both cifically a Docker problem – it’s a startups and incumbents. result of how package managers Over time, it’s likely that work. A build done one day will the distinction between VMs get a given version, and a build and containers (the “run” part of done another time may get a lat- Docker) will become less import- er version, which is why package ant, which will push attention to managers have upgrade facili- the build and ship aspects. The ties. The container abstraction changes here will make the ques- (caring less about the contents tion of what happens to Docker of a container) along with con- much less important than what tainer proliferation (because of happens to the IT industry as a lightweight resource utilisation) result of Docker. is, however, likely to make this a pain point that gets associated with Docker.

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 17 READ ONLINE AT BrianCarpio.com How I Built a Self-Service Automation Platform

Brian Carpio is a technical leader in the IT space with over 15 years of experience Brian designed and supported solutions for multiple Fortune 500 and 100 companies. His specialty is in large scale environments where automation is a must. Brian loves working with new technology like NoSQL and AWS.

The current buzz in the industry is how DevOps is changing the world and how the traditional role of operations is changing. These discussions lead to building self-service automation tools that let developers deploy and support their own code. I want to discuss a little the work I did at a previous company and how it changed the way the company did business.

AWS architecture service). One of our most heated other open standards. Initially, We used AWS as the foundation discussions was over what AWS this wasn’t a well-accepted view- of our platform, but using AWS services we actually wanted to point because many of our de- isn’t the be-all and end-all; if any- support. I was a huge advocate velopers wanted to play with as thing, it presented its own chal- of keeping it simple and sticking many AWS services as they could, lenges to the self-service plat- strictly to EC2-related services but this paid off in the long run. form we ended up designing. (compute, storage, load balanc- ing, and public IPs). The reason Environments Cloud-agnostic was simple: I knew that eventual- AWS doesn’t provide an environ- Amazon provides a wide range ly our AWS expense would grow ments concept, so the first archi- of services from basic EC2 fea- so much that it might eventually tectural decision was on envi- tures like instance creation to make sense to build our own pri- ronments. Having spent the last full blown DBaaS (database as a vate cloud using OpenStack or 15 years working in operations, I

18 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 understood that it was critical to creating an environment of con- stance, we had a v3 branch of design an EC2 infrastructure that sistency. our Puppetfile, which represent- was identical among develop- ed a specific set of Puppet mod- ment, staging, and production. Puppet ules at a specific version; this al- So we made the decision that we We next decided which config- lowed us to work on a v4 branch would create separate VPCs for uration-management system to of Puppetfile that introduced each of these environments. Not use. This was easy, since I wrote breaking changes to the environ- only did the VPC design allow for the prototype application and ment without impacting existing network and routing segregation no one else in my team existed at nodes. A node in our ENC would between each environment, it the time I picked Puppet. Puppet look like this: also allowed developers to have is mature and it allowed me to a full production-like environ- create a team of engineers who classes: ment for development purposes. strictly wrote Puppet modules, which we ultimately made avail- base: ‘’ Security groups able via the self-service REST API. The next decision was how se- nptclient: ‘’ curity groups would work. I’ve Puppet ENC seen complex designs where At the time we built our plat- nagiosclient: ‘’ databases, webservers, product form, Hira was still a bolt-on lines, etc. all have their own se- product, and we needed a way backupagent: ‘’ curity groups, and while I don’t to programmatically add nodes fault anyone for taking such an to Puppet as our self-service mongodb: ‘’ approach, it didn’t really lend automation platform created itself to the type of automation them. Nothing really stood out environment: v3 platform I wanted to design. I as a solution, so we decided to wasn’t willing to give develop- roll our own Puppet ENC, which parameters: ment teams the ability to man- can be found here: http://www. age their own security groups briancarpio.com/2012/08/17/ This allowed our development so I decided there would only puppet-enc-with-mongodb- teams to decide when they actu- be two. I simply created a public backend/ ally wanted to promote existing security group for instances that By using the ENC we creat- nodes to our new Puppet man- lived in the public subnets and a ed, which supported inheritance, ifest or simply redeploy nodes private security group that lived we were able to define a default with the new features. in the private subnets. set of classes for all nodes and This allowed for strict sets then allow specific nodes to have Command line to REST of rules regarding which ports optional Puppet modules. For in- API (80/443) were open to the Inter- stance, if a developer wanted a Our self-service platform started net and which ports would be NodeJS server, they could simply with my working with a single open from the public to the pri- pass the NodeJS module, but by team who wanted to go to the vate zone. I wanted to force all using the inheritance feature of cloud. Initially, I wrote a simple of the applications to listen on a our ENC, we were able to enforce Python module that used boto specific port on the private side. all the other modules like moni- and Fabric to do the following: An amazing thing happens toring, security, base, etc. • Enforce a naming conven- when you tell a developer, “You tion. can push this button an get a Puppet environments • Create an EC2 instance. Tomcat server, and if your app We decided to use Puppet en- • Assign the instance to the runs on port 8080 everything will vironments entirely differently correct environment (dev, work. However, if you want to run than how Puppet Labs intended stg, prd). your app on a non-standard port, them to be used. We used them • Assign the correct security you need to get security approv- to tag specific instances to spe- group. al, and wait for the network team cific versions of our Puppet man- • Assign the instance to a pub- to open the port.” The amazing ifest. By using a branch strategy lic or private subnet. thing is that quickly every ap- for Puppetfile and r10k, we were • Tag the instance with the ap- plication in your environment able to create stable releases plication name and environ- begins to run on standard ports, of our Puppet manifest. For in- ment.

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 19 • Add the node and correct • Instance creation and man- Puppet modules to the Pup- agement pet ENC. • Database deployments • Add the node to Amazon (MongoDB, Cassandra, Elas- Route 53. ticSearch, MySQL) • Register the node with Zab- • Haproxy with A/B deploy- bix. ment capabilities using Con- • Configure Logstash for Kiba- sul na. • Elastic load-balancer man- By using Puppet, the node would agement spin up, connect to the Puppet • Data encryption at rest master, and install the default set • DNS management of classes plus the classes passed • CDN package uploading by our development teams. In • Multi-tenant user-authenti- the steps above, Puppet was re- cation system sponsible for configuring Zabbix, registering with the Zabbix serv- Conclusion er, and configuring Logstash for While the original idea was sim- Kibana. ple, it revolutionized the way my company delivered software to Rapid adoption market. By centrally managing Once development teams were the entire infrastructure with able to deploy systems quickly, Puppet, we had an easy way to bypassing traditional process- push out patches and security es, adoption spread like wildfire. updates. By creating an easy- After about four different teams to-use API, we let development began using this simple Python teams deploy code quickly and module, it became clear that we consistently from development needed to create a full-blown through production. We were REST API. able to maintain infrastructure The good thing about start- best practices with backups, ing with a simple Python module monitoring, etc., all built into the was that the business logic for infrastructure that development interacting with the system was teams could deploy with the defined for a team of real devel- click of a mouse or an API call. opers tasked with building a REST API. The prototype provided the information they needed to cre- ate the infrastructure across the environments, and interact with Zabbix, Kibana, Route 53, etc. It wasn’t long till we were able to implement a simple au- thentication system that used Crowd for authentication and provided team-level ownership of everything from instances to DNS records.

Features Over time, the number of fea- tures grew. Here is the list of features that finally seemed to provide the most value across 70 development teams around the globe.

20 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 READ ONLINE ON InfoQ

Managing Build Jobs for Continuous Delivery

Martin Peston is a build and continuous-integration manager for Gamesys, the UK’s leading online and mobile-gaming company, owners of the popular bingo brand Jackpotjoy and social gaming success Friendzys on Facebook. Martin has 15 years of cross-sector IT experience, including work in aerospace, defence, health, and gambling. He has dedicated around half his career to build management and is passionate about bringing continuous-delivery best practices to software-development companies. In his spare time, Martin is a keen amateur astronomer and is author of A User’s Guide to the Meade LXD55 and LXD75 Telescopes, published by Springer in 2007.

Continuous delivery (CD) takes an evolving product from development through to production. (CI) plays an important part in CD and defines the software development at the beginning of the process.

There is a wealth of information build tool will subsequently run The number of jobs in a describing CI processes but there jobs that compile and test the build tool can range from just a is little information for the many code. The built artifact is then few to perhaps several thousand, different CI tools and build jobs uploaded to a central reposito- all performing various func- that compile and test the code, ry ready for further deployment tions. Fortunately, there is a way which are central to the CI pro- and testing. to manage all of these jobs in a cess. Hence, it is the role of jobs in more efficient manner. In typical CI processes, de- the CI tool to manage the contin- velopers manually build and test uous modifications of the source Automatic creation of source code on their own ma- code, run tests, and manage the build jobs chines. They then commit their logistics of artifact transporta- So why should we set up a facility modifications to a source-con- tion through the deployment to automate the creation of other trol-management system. A pipeline. jobs?

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 21 Build-tool user guides de- Advantages for Some build tools have a built-in scribe how to create build jobs automating the creation feature that can save job configu- but they do not often explain of build jobs ration as a file to any location. how to manage them in any There are several advantages of Since the job configuration great detail, despite the tools’ using a master build job to au- can be saved as a file, it can be ability to do so. tomatically create multiple jobs stored in a configuration-man- Normally, the developer in order to facilitate the CI build agement (CM) system. Any will know how the application is process: change in job configuration can built and so takes sole responsi- 1 All build job configurations be recorded either by modifying bility for the creation and config- are created from a set of job the file directly and then upload- uration of the build job. However, templates or scripts. These ing the file to the build tool or by this process has some disadvan- can be placed under config- modifying the job configuration tages: uration control. Creation of a from the build tool, saving the 1 Software architectural con- new job is then a simple mat- file, and uploading it manually to straints: Applications are ter of using the template or the CM system. The method car- likely to be built differently running the scripts. This can ried out depends on how easy it from one another, depend- cut job-creation time signifi- is to access the job-configuration ing upon architecture. One cantly (many minutes to a file directly from the build tool. developer may interpret how few seconds). Job configura- Another important reason an application is built differ- tion changes are made easi- for storing job configuration in a ently from another develop- er; configuration changes to CM system is that in the event of er and so the build configu- the build-job template en- a catastrophic failure that loses ration will slightly differ from sure that all subsequent new all job configurations, the build one job to another. Hence, jobs will inherit the changes. system can recover relatively a large number of jobs be- 2 Consistency with all existing quickly and restore all jobs, build come unmaintainable if they build-job configurations im- logs, and job history to their last are all configured differently. plies that globally updating known good states. 2 Human factor: Manually all configurations via tools creating a job introduces or scripts is more straightfor- Point 2: Job maintenance the risk of making mistakes, ward than with a cluttered Maintaining a thousand jobs of especially if a new job is cre- inconsistent system. different configurations would ated by copying an existing 3 The developer does not nec- be a major headache, which is job and then modifying that essary need detailed knowl- why it is so important to stan- job. edge of the build tool to cre- dardise job configurations. If a 3 Job timeline control: There ate a build job. required change impacts a large is generally no record kept 4 The ability to automatically number of similar jobs, then writ- of job configuration for ev- create and tear down build ing scripts or creating special ery build, so if a job config- jobs is part of an agile, CI- tasks to modify these jobs would uration is changed, it could build lifecycle. be straightforward. break previous builds. Let’s discuss these points. However, having a thou- Because of the points above, if sand jobs in a build system is not developers are left to create build Point 1: Configuration control the best strategy for CI. We will jobs without a consistent meth- of job configuration discuss this further under Point 4. od for creating jobs, then they It is good practice that every are going end up with a poorly component of a CI system be Point 3: Developer control maintained CI build system and placed under configuration con- Although developers are encour- a headache to manage. This de- trol. That includes the underlying aged to work autonomously, feats the purpose of a CI system: build systems and the build jobs, building their applications using being lean and maintainable. not just the source code. a centralised build system, they Most build tools can create, Most build tools store need to work within guidelines maintain, and back up build jobs build-job configuration as a file outlined by the build process. automatically, usually via API on the master system. These files A developer will expect fast scripts. However, these features, are set to a common XML format feedback on their code changes despite mention in the build- and are normally accessible via and will not want to waste time tool user manual, are often over- the front-end interface directly messing around with build tools looked and not implemented. through a REST API of some kind. and job configuration. Provid-

22 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 ing developers with a one-but- • Run integration tests ton self-service-style solution to Best practices aside – i.e. develop Consistent build automatically set up build jobs on baseline as much as possible, will help them achieve their de- hence few or no development job configuration velopment task objectives more branches – software-develop- quickly. ment teams generally follow the tasks outlined above by running leads to easier job Point 4: A lean CI system a suite of build jobs in the build Although software-development tool. maintenance and teams may adopt agile methods, This is where automatic cre- the build tools they work with are ation of jobs adds huge value to not necessarily used in the same the software-development pro- drives consistent way. A properly configured build cess. If a developer wanted to tool should not contain thou- manually create four or five jobs development sands of long-lived build jobs. to manage all of the develop- Ideally, jobs should be created on ment tasks in day-to-day opera- demand as part of a CI lifecycle. tions, it would take a significant practices across a As long there is the means to rec- amount of time and effort. Com- reate any build job from any his- pare this to automatically creat- company torical job configuration and that ing jobs in a fraction of the time build relics are retained (i.e. logs, at a push of a button. artefacts, test reports, etc.), only a handful of jobs should exist in When to implement the build tool. a self-service job- Of course, an on-demand creation solution job creation/teardown solution Implementing a self-service solu- may not be an achievable goal, tion for creating build jobs is ap- but the point is that the number plicable in the following cases: of jobs in a build tool should be 1 Initialisation of new green- kept to a manageable level. field projects (those that have never been built be- Day-to-day fore). development 2 Projects that already exist Build management governs the but never had any build jobs tools and processes necessary set up for them – i.e. building to assist developers in day-to- manually from a developer’s day activities with regards to own machine. build and CI. Utilities can be set There are also cases where imple- up to help development teams menting self-service job-creation to create build jobs and provide may not apply: flexibility in job configuration 1 If a product only has a few but while remaining governed jobs – such as with large, by software-development best monolithic applications – practices. then it may be considered not of value to the business The CI job suite to spend considerable time Consider typical tasks that devel- and effort to design, test, opers carry out during day-to- and roll out an automated day development: create-job process for just • Build a baseline. those few projects. However, • Release the baseline. as the business expands, the • Build a release branch (often product architecture will be- combined with releasing the come more complex as more baseline). development streams are • Create a development set up to manage the extra branch and build the branch. workload, hence the number

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 23 Diagram 1: Hudson job-creation workflow.

of jobs in the build tool will ciple should be the same for For clarity, I will talk only inevitably increase. building applications for any lan- about Hudson but the informa- 2 Projects with existing jobs guage, such as Java, .NET, etc. tion is equally relevant for Jen- may have issues. Consider Introducing the job-cre- kins. two scenarios of dealing with ation build form The Hudson CI tool has existing jobs in the build tool: The build form is a means to become popular among build • Delete the existing jobs pass information from the build managers looking to minimise and recreate them with the tool directly to another feature their development resource job-creation utility, thereby or back-end scripts that perform budget and who do not want to losing all historical build in- the job-creation tasks. be tied down with expensive li- formation and logs. Ideally, as much project cences. Hudson defines its build • Keep the existing jobs and information should be provid- job-configuration in XML format. attempt to modify their con- ed up front. This will reduce any Typically, this XML file is acces- figuration to match the job later effort required to add extra sible from the top-level page of configuration created by the configurations to newly created the job URL, i.e. http://hostname/ job-creation utility. build jobs. Information such as job-name/config.xml In any case, there is always going project title, location of source Most of the sections in the to be some overhead with main- code, and specific build switch- config file are self-explanatory. taining legacy build jobs. es are normally required. A push Sections are added by plugins in of the “Build” button creates the the config file only when they are Setting up self-service jobs just a few moments later. used in the Hudson GUI. job creation in a build These config files can be tool Build-tool made into templates by replac- We have talked about the advan- implementation ing specific project information tages of automatically creating Two tools with which I have set in them with tokens. Several build jobs, so now let’s discuss up automated job creation are templates can be used to carry how we can implement such a Hudson and AnthillPro 3. out different tasks, e.g. checkout feature in a build tool. The prin- Hudson/

24 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 Figure 1: Hudson create-job form.

FIgure 2: Hudson job view.

Figure 3: AnthillPro job view. of source code from different An example of a Hudson has “create job” privileges, SCMs such as Subversion or Git. API create-job upload command otherwise the upload will fail Now we need to link the cre- is shown below: authentication. ate-job build form to the scripts. curl –H Content-Type:applica- 2 If Hudson tabs are used to Diagram 1 shows the workflow. tion/xml –s –data @config.xml view specific groups of jobs, The build form passes the ${HUDSONURL}/create- it will not be possible to au- information to the scripts, which Item?name=hudsonjobname tomatically place any of the substitute the tokens in the tem- (see Figure 1) newly created jobs under plates with the relevant infor- The jobs can be manually those tabs, except for the ge- mation. The config files are then added to a new view. (see Figure neric “all” tab (or any tab that uploaded to the Hudson tool via 2) uses a regex expression). The an API. More details about the There are a few points to Hudson master-configura- API can be found in Hudson by note: tion file will require updating adding the string “api” to the URL 1 If the Hudson security sec- as well as a manual restart of in the browser, i.e. http://host- tion is configured, ensure the Hudson interface. Once name/job-name/api. that the “anonymous” user the jobs are created, they can

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 25 001 002 003 004 005 006 007 008 009 010 011 ... 012 ... 013 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 ... 032 ... 033 034 035 036 037 038 039 040 041 042 curl -H Content-Type:application/xml -s --data @config.xml ${hudson. createItemUrl}${hudsonjobname}-${job} 043 044 045 046 047 048 049 050 051 052 Code 1

26 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 001 002 003 004 Builds $POM.ARTIFACTID 005 ... 006 ... 007 008 009 010 $SCM.PATH/trunk 011 . 012 infinity 013 false 014 015 016 017 $BUILD.FARM 018 ... 019 ... 020 021 022 $MVN.GOALS 023 $MVN.VERSION 024 false 025 026 027 ... 028 ... 029

Code 2: CI-build-trunk-config.xml Hudson job template.

Figure 4: AnthillPro create-workflow form.

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 27 then be manually added to the relevant tab if so desired. Scripts can be written in any language, such as Ant, Perl, Groovy, etc., so it is a relatively straight- forward set of tasks to create scripts in order to carry out the steps outlined above. A typical Ant script: Code 1 Code 2 is a typical tokenised Hudson job template:

AnthillPro AnthillPro is a licensed build and CI tool from UrbanCode (now part of IBM) that provides the user almost complete custom control of build jobs via a GUI and an extensive API library. The latest incarnation from UrbanCode is uBuild, which is essentially a cut-down version of AnthillPro and is geared primarily towards the CI building of products whereas its parent product was originally designed to manage an entire CD pipeline. uBuild does not have the API capability that AnthillPro has but it is possible to create a plugin that can do a similar thing. Build jobs in AnthillPro are called work- flows, which essentially are pipelines of individu- al job tasks executed in a specific order (in series or parallel). AnthillPro’s API-scripting language, Bean- Shell, is based on standard Java methods. BeanShell scripts are stored within the Ant- hillPro graphic user interface and called from within a job task. The latest version of AnthillPro allows developers to custom-build plugins with any programming language. An AnthillPro job task is created using API calls that essentially construct all the necessary functions of that workflow. The script can be par- ticularly long, as every aspect of the workflow configuration has to be populated with data. Just like Hudson, a build form collects build-job information and executes a master workflow, calling on the BeanShell scripts to create the rel- evant workflow. (Figure 3 & 4) A typical Java BeanShell script: Code 2 Note: It is also possible to create a self-con- tained package of Java BeanShell scripts such as a jar file, which is placed in a folder on the Ant- hillPro application server. Direct calls to classes within the jar file can then be made from a shell task within the application. This works well, es- pecially because the scripts can be uni-tested before being applied in a live development en- vironment.

28 001 private static Project createProject(User user) throws Exception { 002 // get the values from the buildlife properties 003 String groupId = getAnthillProperty(“GROUPID”); 004 String artifactId = getAnthillProperty(“ARTIFACTID”); 005 String build_farm_env_name = “linux-build-farm”; 006 007 String branchName = “branchName”; 008 009 Workflow[] workflows = new Workflow[3]; 010 // Set up Project 011 Project project = new Project(artifactId + “_” + branchName); 012 013 // determine whether the project already exists and is active 014 boolean isProjectActive; 015 try { 016 Project projectTest = 017 getProjectFactoryInstance(artifactId + “_” + branchName); 018 isProjectActive = projectTest.isActive(); 019 } catch (Exception err) { 020 isProjectActive = false; 021 } 022 023 if (!isProjectActive) { 024 project.setFolder(getFolder(groupId, artifactId)); 025 setLifeCycleModel(project); 026 setEnvironmentGroup(project); 027 setQuietConfiguration(project); 028 029 String applications = getAnthillProperty(“APPS”); 030 // create project properties 031 createProjectProperty(project, “artifactId”, artifactId, false, false); 032 // set the server group for the workflow and add environment properties 033 addPropertyToServerGroup(project, build_farm_env_name, applications); 034 035 project.store(); 036 037 ... 038 ... 039 040 // Create the CI Build workflow 041 String workflowName = “CI-build”; 042 workflows[0] = createWorkflow( 043 project, 044 workflowName, 045 perforceClientFor(branchName, groupId, artifactId, workflowName) + buildLifeId, 046 perforceTemplateFor(branchName, groupId, artifactId, workflowName), 047 “${stampContext:maven_version}”, 048 build_farm_env_name, 049 “CI-build Workflow Definition” 050 ); 051 052 // add trigger url to commit-build branch 053 addRepositoryTrigger(workflows[0]); 054 055 // Create the Branch workflow 056 workflowName = “Branch”; 057 workflows[1] = createWorkflow(

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 29 058 project, 059 workflowName, 060 perforceClientFor(branchName, groupId, artifactId, workflowName) + buildLifeId, 061 perforceTemplateFor(branchName, groupId, artifactId, workflowName), 062 “From ${stampContext:maven_version}”, 063 build_farm_env_name, 064 “Branch Workflow Definition” 065 ); 066 067 // Create the Release workflow 068 workflowName = “Release”; 069 workflows[2] = createWorkflow( 070 project, 071 workflowName, 072 perforceClientFor(branchName, groupId, artifactId, workflowName) + buildLifeId, 073 perforceTemplateFor(branchName, groupId, artifactId, workflowName), 074 “${stampContext:maven_release_version}”, 075 build_farm_env_name, 076 “Release Workflow Definition” 077 ); 078 079 ... 080 ... 081 082 } else { 083 // project already exixts 084 String msg = “Project “ + artifactId + “_” + branchName + “ already exists - check project name and pom”; 085 throw new RuntimeException (msg); 086 } 087 return project; 088 } Code 3: Sample CreateProject Java BeanShell script.

Other tools matically create and tear down should be adopted as part of As long there exists the ability to jobs on a build tool, so I cannot everyday software develop- create jobs via an API or a tem- detail the technical challenges ment. plate feature, the implementa- to implement such a task. How- • Automatic creation of jobs tion described here can be simi- ever, everything that has been saves time and allows the larly applied to other build tools discussed here can lead to imple- developer to get on with such as Go, TeamCity, and Bam- menting such a solution. In fact, more important tasks. boo to name but a few. Atlassian’s Bamboo build tool • Consistent build-job config- Ultimate build-tool agility: can detect branch creation from uration leads to easier job Implementing the lean build- a main baseline and automati- maintenance and drives con- tool philosophy cally create an appropriate job to sistent development practic- An advantage mentioned build it. es across a company. earlier in this article is automat- • Automatic job creation can ically creating on-demand CI Summary ultimately lead to a true agile projects and tearing them down We have discussed how to set up adoption of software devel- after completion. This would sig- facilities to create other jobs that opment right down to the nificantly reduce the amount of form part of the continuous de- build-tool level. build jobs in the build tool. livery strategy for a business: I have yet to implement • An automated approach to an agile, lean method to auto- creating jobs in a build tool

30 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 READ ONLINE ON SimplicityItself.com

Service Discovery

David Dawson is CEO of Simplicity Itself, a UK-based consultancy that specialises in understanding and building simple software that thrives on change. After leading Simplicity Itself in the building of several large microservices systems and applying DevOps in their construction and design, David now applies his experience to the development of Muon, a new open-source programming model for microservices.

When building microservices, you have to distribute your application around a network. It is almost always the case that you are building in a cloud environment, and often using immutable infrastructure. Ironically, this means that your virtual machines or containers are created and destroyed much more often than normal, as this immutable nature means that you don’t maintain them.

These properties together This concept is one of the managers, have some form of mean that your services need to key underpinnings of micros- service discovery. be reconfigured with the loca- ervice architecture. Attempting There are current- tion of the other services they to create microservices without ly several key contenders to need to connect to. This recon- a service-discovery system will choose from: ZooKeeper, Con- figuration needs to be able to lead to pain and misery, as you sul, etcd, Eureka, and rolling your happen on the fly, so that when will in effect be working as a own solution. a new service instance is created, manual replacement. the rest of the network can auto- As well as the standalone ZooKeeper matically find it and start to com- solutions presented here, most ZooKeeper is an Apache project munication with it. This process platforms, whether full PaaS that provides a distributed, even- is called service discovery. or the more minimal container tually consistent, hierarchical configuration store.

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 31 implementation of master selec- and be discoverable in ZooKeep- tion is robust and well behaved. er. If you do choose to use Zoo- Deployment-wise, Consul Keeper, investigate the Netflix agents are deployed on the sys- OSS projects, starting with Cura- tems that services are running tor, and only use bare ZooKeeper on, and not in a centralised fash- if they don’t fit your needs. ion. Since ZooKeeper is ma- This is a newer product and ture and established, there is a one that Simplicity Itself likes a large ecosystem of good-quality lot. We recommend it if you are (mostly!) clients and libraries to able to adopt it. enrich your projects. As with the other strongly consistent systems in the list, care Consul must be taken that you under- ZooKeeper originated in the stand the implications of adopt- world of Hadoop, where it was ing it, including understanding built to help in the maintenance its potential failure modes. of the various components in If you would like something a Hadoop cluster. It is not a ser- similar that chooses availability vice-discovery system per se, but rather than consistency, investi- is instead a distributed config- gate the related serf project. The uration store that provides no- serf library serves as the basis tifications to registered clients. of Consul, but has chosen dif- With this, it is possible to build a Consul is a peer-to-peer, strong- ferent data guarantees. It is no- service-discovery infrastructure ly consistent data store that uses where near as fully featured, but although every service must ex- a gossip protocol to communi- can handily survive a split-brain plicitly register with ZooKeeper, cate and form dynamic clusters. scenario and reform afterwards and the clients must then check It is based on the serf library. without any ill effects. in the configuration. It provides a hierarchical key-val- Netflix has invested a lot of ue store in which you can place etcd time and resources in ZooKeeper data and register watches to be so a significant number of Netflix notified when something chang- OSS projects have some Zoo- es within a particular key space. Keeper integration. In this, it is similar to ZooKeeper. ZooKeeper is a well-under- As opposed to ZooKeeper stood clustered system. It is a and etcd, however, Consul im- consistent configuration store, so plements a full-service discovery a network partition will cause the system in the library, so you don’t smaller side of the partition to need to implement your own shut down. For that reason, you or use a third-party library. This must choose whether consisten- includes health checks on both cy or availability is more import- nodes and services. ant to you. It implements a DNS server As an HTTP-accessible key-value If you do choose a consis- interface, allowing you to per- store, etcd is similar in concept tent system such as ZooKeeper form service lookups using the to ZooKeeper and the key-value for service discovery, you need to DNS protocol. It also allows cli- portion of Consul. It functions as understand the implications for ents to run as independent pro- a distributed, hierarchical config- your services. You have tied them cesses and can register/monitor uration system, and can be used to the lifecycle of the discovery services on their behalf. This re- to build a service-discovery sys- system and have exposed them moves the need to add explicit tem. to any failure conditions it may Consul support to your applica- It grew out of the CoreOS have. You should not assume tions. This is similar in concept to project, which maintains it. The that “consistent” means free from the Netflix OSS Sidecar concept tool recently achieved a stable failure. ZooKeeper is among the that allows services with no Zoo- major release. older cluster managers, and con- Keeper support to be registered If you are primarily using sensus (pun intended) is that its HTTP as your communication

32 Advanced DevOps Toolchain // eMag Issue 28 - May 2015 mechanism, etcd can’t be easily the ability to continue working if • It must notify you of services beaten. It provides a well-distrib- the service-discovery infrastruc- starting and stopping. uted, fast, HTTP-based system, ture fails. Building your own discovery ser- and has query and push notifica- Server-wise, Eureka has also vice should not be taken lightly. tions on change via long polling. chosen availability rather than If you need to do it, we at Sim- consistency. You must also be plicity Itself recommend building Eureka aware of the implications of this a system that values availability choice as it directly affects your rather than consistency. These application. Primarily, this mani- are significantly easier to build, fests as a potentially stale or par- and make it more likely that you tial view of the full data set. This will build something functional. is discussed in the Eureka docu- We recommend that you mentation. use some existing message in- Eureka is now the quickest frastructure and broadcast noti- to get started with Spring proj- fications on service status. Each ects because of the investment service caches the latest infor- the Spring team has made in mation from the broadcasts and adopting the Netflix OSS compo- uses that as a local set of ser- nents via the Spring Cloud sub- vice-discovery data. This has the projects. potential to become stale, but we’ve found that this approach Roll your own scales reasonably well and is easy Eureka is a mid-tier load balanc- to implement. er built by Netflix and released If you do require consisten- as open source. It is designed to cy, using some consistent data allow services to register with a store could serve as the basis Eureka server and then to locate for a distributed configuration each other via that server. system that can be used to build Eureka has several capabil- service discovery. You will also ities beyond the other solutions want to emit notifications on presented here. It contains a status changes. You should real- built-in load balancer that, al- One major point to note in the ise, though, that building a con- though fairly simple, certainly above systems is that they all sistent, distributed system is ex- does its job. Netflix states that require some extra service-dis- ceptionally hard to get right, and they have a secondary load-bal- covery infrastructure. If you can very easy to get subtly wrong. ancer implementation internally accommodate that, it is better Overall, we do not recom- that uses Eureka as a data source to adopt one or more of the sys- mend this strategy but it is cer- and is much more fully featured. tems above for service discovery. tainly possible. This hasn’t been released. If you can’t adapt to that, If you use Spring for your you will have to create your own projects, Spring Cloud is an inter- discovery solution within your esting project to look into. It au- existing infrastructure. tomatically registers and resolves The basis of this will be that: services in Eureka. • Services must be able to no- Like all Netflix OSS projects, tify each other of their avail- Eureka was written to run on the ability and supply connec- AWS infrastructure first and fore- tion information. most. While other Netflix OSS • Periodic updates to the re- projects have been extended to cords should strip out stale run in other environments, Eure- information. ka does not appear to be moving • It must easily integrate with in that direction. your application infrastruc- We at Simplicity Itself very ture, often using a standard much like the relation between protocol such as HTTP or client and server in the Eureka DNS. design, as it leaves clients with

Advanced DevOps Toolchain // eMag Issue 28 - May 2015 33 PREVIOUS ISSUES

QCon London 2015 Report

This eMag provides a round-up26 of what we and some of our attendees felt were the most important sessions from QCon London 2015 including Microservices, Con- tainers, Conway’s Law, and properly implementing agile practises.

Next Generation HTML5 and JavaScript

Modern web developers have25 to juggle more con- straints than ever before; standing still is not an option. Technical Debt We must drive forward and this eMag is a guide to get- and Software ting on the right track. We’ll hear from developers in the Craftsmanship trenches building some of the most advanced web ap- plications and how to apply what they’ve learned to our 27 own work. This e-mag brings together a number of authors who have addressed the topic on InfoQ and suggests ways to identify, monetize, and tackle technical debt in your products. Mobile - Recently New Technology and Already a Commodity?

This eMag discusses some familiar24 and some not too fa- miliar development approaches and hopefully will give you a helping hand while defining the technology stack for your next mobile application.