Learning CFEngine 3 Diego Zamboni Beijing • Cambridge • Farnham • Köln • Sebastopol • Tokyo Learning CFEngine 3 by Diego Zamboni Copyright © 2012 Diego Zamboni. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/institutional sales department: (800) 998-9938 or [email protected]. Editors: Andy Oram and Mike Hendrickson Cover Designer: Karen Montgomery Production Editor: Dan Fauxsmith Interior Designer: David Futato Proofreader: O’Reilly Production Services Illustrator: Robert Romano Revision History for the First Edition: 2012-03-16 First release 2012-11-09 Second release See http://oreilly.com/catalog/errata.csp?isbn=9781449312206 for release details. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Learning CFEngine 3 and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information con- tained herein. ISBN: 978-1-449-31220-6 [LSI] 1352479361 Table of Contents Foreword ................................................................... vii Preface ..................................................................... xi 1. Introduction ........................................................... 1 How to Achieve Automation 3 Home-Grown Scripts 3 Specialized Tools for Automation 4 Why CFEngine? 6 A Brief History of CFEngine 7 Versions of CFEngine 8 2. Getting Started with CFEngine ........................................... 11 Installing CFEngine 11 Installing the Community Edition from Source 12 Installing the Community Edition from Binary Packages 15 Installing the Commercial Edition 15 Finishing the Installation and Bootstrapping 16 Auxiliary Files 19 Your First CFEngine Policy 19 3. CFEngine Basics ........................................................ 25 Basic Principles 25 Desired-State Configuration 25 Basic CFEngine Operations 26 Promise Theory 27 Convergent Configuration 29 CFEngine Components 29 A First Example 32 CFEngine Policy Structure 34 Data Types and Variables in CFEngine 35 iii Classes and Decision Making 40 Containers 44 Normal Ordering 53 Looping in CFEngine 56 Thinking in CFEngine 58 Clients and Servers 59 CFEngine Server Configuration 61 Updating Client Files from the Server 63 CFEngine Remote Execution Using cf-runagent 65 CFEngine Information Resources 67 Manuals and Official Guides 67 CFEngine Standard Library 68 CFEngine Solutions Guide 68 CFEngine Design Center 69 Community Forum and IRC channel 69 CFEngine Bug Tracker 69 Other Community Resources 69 Recommended Reading Order 70 4. Using CFEngine ........................................................ 71 Initial System Configuration 71 Editing /etc/sysctl.conf 71 Editing /etc/sshd_config 80 Editing /etc/inittab 85 Configuration Files with Variable Content 88 User Management 93 Software Installation 97 Package-Based Software Management 97 Manual Software Management 101 Using CFEngine for Security 109 Policy Enforcement 109 Security Scanning 114 5. CFEngine Tips, Tricks, and Patterns ...................................... 121 Hierarchical Copying 121 Passing Name-Value Pairs to Bundles 128 Setting Default Values for Bundle Parameters 131 Using Classes as Configuration Mechanisms 132 Generic Tasks Using Lists and Array Indices 135 Defining Classes for Groups of Hosts 138 Controlling Promise Execution Order 140 iv | Table of Contents 6. Advanced Topics ...................................................... 143 Setting Up Multiple CFEngine Environments 143 Using a Version-Control System to Separate Environments 147 Flow of Development and Deployment 148 CFEngine Testing 149 Behavioral Testing for CFEngine Policies 150 Unit Testing for CFEngine Policies 150 Where to from Here? 156 A. Editing CFEngine 3 Configurations in Emacs ............................... 159 B. Editing CFEngine 3 Configurations in Vim ................................. 165 Table of Contents | v Foreword The history of “Unix” system configuration has been a fascinating ride that took us from shell scripting to sophisticated knowledge-oriented tools. I still recall arriving in San Diego in 1997 for the USENIX/LISA conference, just three years after releasing CFEngine to the wider world as a GNU Free Software distribution. I walked through the door from conference registration and the first person I met looked at my badge and said: “Hey, you’re Mark Burgess—you wrote CFEngine!” That was my first exposure to the power of community. Free Open Source Software (FOSS) was a kind of Berlin Wall moment for the software industry, removing the barriers to contributing innovative ideas that had been closed off by fearful corporate protectionism. Perhaps ironically, “free software” was the door- opener to innovation that enabled Internet commerce to take off—transforming Ri- chard Stallman’s vision of “free speech” into a definite focus on “free beer,” but with the importance of community social networks strongly emphasized. To me, what was important about FOSS was that it enabled research and development to flourish and find a willing audience, all without anyone’s approval. For CFEngine, this was central to overcoming limitations steeped in the past. When I began writing CFEngine in 1993, inspired by colleagues at Oslo University, the main problem lay in handling a diversity of operating systems. There were many more flavors of Unix-like OS back then, and they were much more different than they are today. Writing any kind of script was a nightmare of exception logic: “If this is SunOS 4.x or Ultrix, but not SunOS 4.1 or anything at the Chemistry department, and by the way patch 1234 is not installed, then...” Such scripts appealed to a generation of “Large Installation System Administrators,” who had deep system experience and basic programming skills. Alas, in such a script, you couldn’t see the intention for the logic, so many scripts were thrown away and rewritten in the latest cool scripting language each time someone arrived or left. It was a time-wasting chaos. The separation of “intended outcome” from the detailed imperative coding was the first purpose of a specialized language for system administration, i.e., making infra- vii structure documentation of intent rather than unreadable code—or as declarative pro- grammers would say, the separation of “what” from “how.” As a theoretical physicist, in postdoctoral purgatory, instinct moved me to look into the scientific literature of the subject of system management, and I discovered that there was very little work done in the field of host configuration. As I left the conference in 1997, I got sick on the plane, and this gave me an idea. A year later, I went back to the LISA conference and wrote down a research manifesto for “autonomic self-healing systems” called Computer Immunology. IBM’s autononomic computing initiative fol- lowed a few years later. Those were heady days of CFEngine history, filled with ex- citement and discovery of principles like “convergence” and “adaptive locking.” At LISA 98, I presented “Computer Immunology” in one hall of the conference while Tom Perrine (then of the San Diego Supercomputing Center, later LOPSA president) opened his talk in the next room with the flattering words: “I owe Mark Burgess more beer than I can afford...” And thus the partnership between science and community was begun. CFEngines 1 and 2 took the world by storm. No one really knows how many agents are running out there, but it runs into the many millions. A large covert community still thrives behind the scenes, making little noise. Recently, a large internet retailer indicated a million computers running CFEngine 2, saying: “Well, it just works.” Sim- ilar stories abound. Even so, CFEngine had rough edges, and we saw plenty of room for improvement. As the Web 2.0 companies were emerging in the 2000s, other tools began to emerge for configuration, bringing back the idea of “Script It Yourself” to engage a generation of web programmers impatient with the idea of system administration getting in the way of more agile methods. Software packaging developed into an important simplification of the configuration—but much too simplistic to support the required competitive differentiation in an application-driven era of IT. From this tension, the idea of DevOps began to emerge and configuration moved back in the direction of custom coding, aided by “easy language frameworks” like Ruby. By this time, I had developed a new model for CFEngine that captured its famous distributed autonomy, and had brought CFEngine its documentable scalability and security properties. This model came to be known as Promise Theory, and as I devel- oped and tested the idea from 2004-2007 I realized that the challenge was not at all about scripting or programming, but really about knowledge and documentation (“The