C u s t o m i z i n g Chef GETTING THE MOST OUT OF YOUR INFRASTRUCTURE AUTOMATION Jon Cowie www.it-ebooks.info Customizing Chef Customizing Take advantage of Chef’s highly customizable design to tackle specific Whether you are new automation issues that you can’t solve by simply using Chef’s tools, “ resources, and services out of the box. With this practical guide, you’ll to Chef or have been examine the internal structure of this configuration management using it for years, this technology and learn where, how, and why to add custom code. book has something Author Jon Cowie not only provides code snippets and practical advice for you! Approachable, for customizing Chef, but also helps you determine case by case whether C u s t o m i z i n g practical advice, working it’s in your best interests to customize at all. If you have intermediate-to- advanced Chef experience, this book is indispensable. code, and best practices can all be found in this ■ Explore Chef’s typical setups and why you’d want to delightful book, which customize them should be considered ■ Learn the Ruby concepts needed to write customizations required reading and ■ Customize the Chef run process and explore Chef’s internal a ready reference for classes anyone who is serious Chef ■ Chef Gain more insight into your Chef runs through handlers and Chef’s event stream about automating GETTING THE MOST OUT OF YOUR infrastructure with Chef. ■ Level up your Chef recipes by writing your own resources ” INFRASTRUCTURE AUTOMATION and providers —Nathen Harvey Community Director at Chef and Co-host ■ Create and extend plugins for Knife—Chef’s command-line of the Food Fight Show podcast tool ■ Interact with the Chef API to write scripts and reports ■ Learn how to contribute customizations to the Chef community Jon Cowie is a senior operations engineer at Etsy, where he solves interesting problems through his love of both coding and operations. He’s created and open- sourced several projects, including the Chef tools knife-spork and knife-preflight. Cowie SYSTEM ADMINISTRATION Twitter: @oreillymedia facebook.com/oreilly US $39.99 CAN $41.99 ISBN: 978-1-491-94935-1 Jon Cowie www.it-ebooks.info Customizing Chef Jon Cowie www.it-ebooks.info Customizing Chef by Jon Cowie Copyright © 2014 Jonathan Cowie. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/ institutional sales department: 800-998-9938 or [email protected]. Editors: Courtney Nash and Brian Anderson Indexer: Wendy Catalano Production Editor: Nicole Shelby Cover Designer: Ellie Volckhausen Copyeditor: Rachel Head Interior Designer: David Futato Proofreader: Gillian McGarvey Illustrator: Rebecca Demarest September 2014: First Edition Revision History for the First Edition: 2014-08-26: First release See http://oreilly.com/catalog/errata.csp?isbn=9781491949351 for release details. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Customizing Chef, the picture of a ruffed grouse, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. ISBN: 978-1-491-94935-1 [LSI] www.it-ebooks.info This book is dedicated to my uncle, David Shrimpton, without whose encouragement and guidance I would likely never have entered this profession. He was a great friend and mentor to all of his students at the University of Kent as well as to myself, and is greatly missed by all who knew and loved him. www.it-ebooks.info www.it-ebooks.info Table of Contents Foreword. xiii Preface. xv Part I. Chef and Ruby 101 1. Introduction. 3 What Is Configuration Management? 3 So Why Chef? 4 Think Critically 7 Meet AwesomeInc 8 Criteria for Customization 10 How Do We Find Out When Our Chef Runs Are Failing, and Why? 11 How Do We Simplify Our Recipes to Reduce the Amount of Replicated Code? 12 How Do We Stop Our Developers and Ops Staff from Treading All over Each Other’s Changes? 12 State of the Customization Nation 13 Chef Supermarket 13 Development Tooling 13 Workflow Tooling 14 Knife Plugins 15 Handlers 16 Recipes and Resources 17 Chef Installation Types and Limitations 18 chef-solo 18 Open Source Chef 19 Local Mode 20 Private Enterprise Chef 20 Hosted Enterprise Chef 21 v www.it-ebooks.info Prerequisites 22 Knife 22 Nodes and Clients 22 Cookbooks, Attributes, Roles, Environments, and Data Bags 22 Chef Search 23 Ruby 23 Assumptions 23 2. Just Enough Ruby to Customize Chef. 25 Ruby Is Object-Oriented 26 Ruby Is Dynamically Typed 27 Classes 30 Getter and Setter Methods 31 Variable Scoping 33 Local Variables 34 Class Instance Variables 35 Inheritance 36 Modules 37 Modules as Namespaces 38 Modules as Mixins 39 Using Other Classes and Modules 41 Local Classes 41 RubyGems 42 Built-in Classes 43 When Things Go Wrong 44 Exceptions 44 Handling Exceptions 46 Defining Custom Exception Types 48 Tying It All Together 49 File Operations 50 HTTP Requests 52 Summary 55 3. Chef Internals. 57 Chef Architecture 58 Chef Client Tools 59 Chef Server 60 Anatomy of a Chef Run 62 Get Configuration Data 63 Authenticate/Register 64 Load and Build Node Object 65 Synchronize Cookbooks 66 vi | Table of Contents www.it-ebooks.info SetUp RunContext 66 Load Cookbook Data 68 Converge Node 69 Finalize 69 Dry-Run and Why-Run 70 The Problem with Dry-Run 71 Why-Run Mode 71 Usefulness of Why-Run 72 Using the Source 74 Getting the Chef Source Code 74 Chef Source Structure 75 Tracing a chef-client Run 77 Execute the chef-client Command 78 Run the Real chef-client Script 79 The Chef::Application::Client Class 80 The Chef::Application::Client run_chef_client Method 83 The Chef::Client Class 84 The Chef::Client Class do_run Method 86 Summary 91 Part II. Customizing Chef Runs 4. Extending Ohai. 95 Introduction 96 Ohai Attribute Collection 96 The Ohai Source Code 97 Ohai Example 1: Plugin Skeleton 97 Testing and Running Ohai Plugins 99 Testing Using irb 99 Running Using Chef 101 Ohai Example 2: Adding More to the Plugin 101 The Mash Object 102 Multiple collect_data Methods 103 Running example2.rb 105 Ohai Example 3: Multilevel Plugins 106 Summary 111 5. Creating Handlers. 113 Preparing a Test Environment 114 Create the Test chef-client Configuration 114 Create a Test Cookbook 115 Table of Contents | vii www.it-ebooks.info Verify That the Test Environment Works Correctly 116 Introduction to Handlers 117 The Run Status Object 119 Run Status Methods 119 Run Information Methods 120 Run Object Methods 121 Handler Example 1: Start Handler 122 Handler Example 2: Report Handler 126 Handler Example 3: Exception Handler 130 Handlers: Summary and Further Reading 136 6. Extending Chef Run Events. 139 Introduction to the Event Dispatcher 139 Event Dispatcher Initialization 140 Publish Overview 140 Subscribe Overview 141 Creating Formatters 142 Formatter Example 1: Skeleton Formatter 143 Formatter Example 2: Slightly Less Skeletal 144 Formatter Example 3: Custom Event Methods 146 Setting the Default Formatter 149 Formatters: Summary and Further Reading 150 Creating Custom Subscribers 151 Subscriber Example 1: Skeleton Subscriber 152 Subscriber Example 2: Registration with Start Handlers 152 Subscriber Example 3: Custom Event Methods 155 Custom Subscribers: Summary 157 Revisiting AwesomeInc—Which Customization? 158 Summary 161 Part III. Customizing Recipes 7. Definitions and Libraries. 165 Cookbook Structure Revisited 166 Creating a Test Environment 167 Definitions 168 What Is a Definition? 168 Definition Example 1: Skeleton 170 Adding Parameters 172 Definition Example 2: Using Parameters 173 Adding Resources 175 viii | Table of Contents www.it-ebooks.info Definition Example 3: Using Resources 176 Libraries 179 What Is a Library? 179 Library Example 1: Modules and Mixins 181 Library Example 2: Methods in a Custom Namespace 184 Library Example 3: Extending Core Chef Classes 186 Summary and Further Reading 188 8. Lightweight Resources and Providers. 191 Introduction to Resources and Providers 192 Automatically Choosing Providers 193 Resources and Providers—Lightweight Versus Heavyweight 194 Introduction to LWRPs 195 LWRP Example 1: Getting Started with LWRPs 196 Example 1: Creating a Resource 196 Example 1: Creating a Provider 198 Example 1: Using Our LWRP in a Recipe 199 The LWRP Domain-Specific Language 200 Resource DSL 201 Provider DSL 203 LWRP Example 2: Extending the Provider 205 Provider Internals 209 Desired Resource State 210 Current Resource State 212 Identifying and Executing Changes 215 Informing Chef About Updates 216 LWRP Example 3: Native Provider 217 Example 3: Preparing the Resource and Provider 217 Example 3: The load_current_resource Method 218 Example 3: Coding the Actions 220 Summary and Further Reading 224 9. Heavyweight Resources and Providers. 227 Introduction to HWRPs 228 Why Write HWRPs? 228 HWRPs in Cookbooks 229 HWRP Example 1: Getting Started with HWRPs 229 Example 1: Creating a Resource 230 Example 1: Creating a Provider 233 Example 1: Using Our HWRP in a Recipe 234 Example 1: Extending the Provider 235 Advanced HWRP Tricks 240 Table of Contents | ix www.it-ebooks.info Mixing HWRPs and LWRPs 240 Overriding Automatic Provider Resolution 243 Implementing Multi-Provider Resources 247 HWRP Example 2: Extending the package Resource 251 Example 2: Creating a Provider 251 Example 2: Using Our Provider in a Recipe 253 Example 2: Tapping into Chef’s Provider Mapping 255 Summary and Further Reading 258 Part IV.