Network Programmability and Automation SKILLS FOR THE NEXT-GENERATION NETWORK ENGINEER Jason Edelman, Scott S. Lowe & Matt Oswalt Praise for Network Programmability and Automation Jason, Scott, and Matt have been key contributors in educating network engineers about both network automation and Linux networking. They have written and talked extensively about the importance of automation, on how automation impacts network engineers, and on the mechanics of automating networking devices. —Kirk Byers Creator of the Netmiko Python Library Network automation is no longer just a proof of concept: it represents both the present and the future! Network Programmability and Automation provides the needed background for modern engineers, by widening the toolset for more consistent, stable and reliable networks. —Mircea Ulinic Network Systems Engineer, Cloudflare Network automation is not hype anymore; it is a means to do your job faster, more consistently and more reliably. However, network automation is not just a single discipline; it is a collection of protocols, tools, and processes that can be overwhelming to the uninitiated. This book does a great job covering everything you will need to get your automation up and running. —David Barroso creator of NAPALM Network Programmability and Automation Skills for the Next-Generation Network Engineer Jason Edelman, Scott S. Lowe, and Matt Oswalt Beijing Boston Farnham Sebastopol Tokyo Network Programmability and Automation by Jason Edelman, Scott S. Lowe, and Matt Oswalt Copyright © 2018 Jason Edelman, Scott S. Lowe, Matt Oswalt. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com/safari). For more information, contact our corporate/insti‐ tutional sales department: 800-998-9938 or [email protected]. Editors: Virginia Wilson and Courtney Allen Interior Designer: David Futato Production Editor: Colleen Cole Cover Designer: Karen Montgomery Copyeditor: Dwight Ramsey Illustrator: Rebecca Demarest Proofreader: Rachel Monaghan Technical Reviewers: Patrick Ogenstad, Akhil Behl, Indexer: Judy McConville Eric Chou, Sreenivas Makam February 2018: First Edition Revision History for the First Edition 2018-02-02: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781491931257 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Network Programmability and Automa‐ tion, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-491-93125-7 [M] I dedicate this book to all network engineers starting their network automation journey. I sincerely hope it provides each of you with the knowledge needed to further enhance your career. I’d also like to thank Scott, Matt, and the whole O’Reilly team—I know it was a much longer process than we all planned, but we ultimately got through it! Thanks to everyone for making it a reality. Jason Edelman I’d like to dedicate this book to the Lord, who granted me the wisdom and understanding I needed to write this book (Exodus 31:3 NIV). I’d also like to dedicate it to my wife, Crystal, without whose support things like this wouldn’t be possible. Scott S. Lowe I dedicate this book to anyone with a hunger and a passion for learning—every word was written with you in mind. I’d also like to thank my wife Jamie, who keeps me moti‐ vated and upbeat when life gets a little too crazy. Matt Oswalt Table of Contents Preface. xiii 1. Network Industry Trends. 1 The Rise of Software Defined Networking 1 OpenFlow 1 What Is Software Defined Networking? 5 Summary 16 2. Network Automation. 17 Why Network Automation? 18 Simplified Architectures 18 Deterministic Outcomes 19 Business Agility 19 Types of Network Automation 20 Device Provisioning 20 Data Collection 23 Migrations 24 Configuration Management 25 Compliance 25 Reporting 26 Troubleshooting 26 Evolving the Management Plane from SNMP to Device APIs 28 Application Programming Interfaces (APIs) 28 Impact of Open Networking 32 Network Automation in the SDN Era 33 Summary 33 vii 3. Linux. 35 Examining Linux in a Network Automation Context 35 A Brief History of Linux 36 Linux Distributions 37 Red Hat Enterprise Linux, Fedora, and CentOS 37 Debian, Ubuntu, and Other Derivatives 39 Other Linux Distributions 40 Interacting with Linux 40 Navigating the Filesystem 41 Manipulating Files and Directories 46 Running Programs 52 Working with Daemons 55 Networking in Linux 60 Working with Interfaces 60 Routing as an End Host 71 Routing as a Router 75 Bridging (Switching) 77 Summary 83 4. Learning Python in a Network Context. 85 Should Network Engineers Learn to Code? 86 Using the Python Interactive Interpreter 88 Understanding Python Data Types 90 Learning to Use Strings 91 Learning to Use Numbers 100 Learning to Use Booleans 102 Learning to Use Python Lists 105 Learning to Use Python Dictionaries 111 Learning About Python Sets and Tuples 115 Adding Conditional Logic to Your Code 117 Understanding Containment 119 Using Loops in Python 121 Understanding the while Loop 121 Understanding the for Loop 122 Using Python Functions 126 Working with Files 129 Reading from a File 130 Writing to a File 132 Creating Python Programs 134 Creating a Basic Python Script 134 Understanding the Shebang 135 Migrating Code from the Python Interpreter to a Python Script 137 viii | Table of Contents Working with Python Modules 138 Passing Arguments into a Python Script 140 Using pip and Installing Python Packages 141 Learning Additional Tips, Tricks, and General Information When Using Python 143 Summary 149 5. Data Formats and Data Models. 151 Introduction to Data Formats 151 Types of Data 153 YAML 154 Reviewing YAML Basics 155 Working with YAML in Python 158 Data Models in YAML 159 XML 160 Reviewing XML Basics 160 Using XML Schema Definition (XSD) for Data Models 161 Transforming XML with XSLT 163 Searching XML Using XQuery 167 JSON 167 Reviewing JSON Basics 167 Working with JSON in Python 170 Using JSON Schema for Data Models 171 Data Models Using YANG 172 YANG Overview 172 Taking a Deeper Dive into YANG 173 Summary 176 6. Network Configuration Templates. 177 The Rise of Modern Template Languages 178 Using Templates for Web Development 179 Expanding On the Use of Templates 180 The Value of Templates in Network Automation 180 Jinja for Network Configuration Templates 181 Why Jinja? 181 Dynamically Inserting Data into a Basic Jinja Template 182 Rendering a Jinja Template File in Python 183 Conditionals and Loops 185 Jinja Filters 191 Template Inheritance in Jinja 195 Variable Creation in Jinja 196 Summary 196 Table of Contents | ix 7. Working with Network APIs. 199 Understanding Network APIs 200 Getting Familiar with HTTP-Based APIs 200 Diving into NETCONF 204 Exploring Network APIs 213 Exploring HTTP-Based APIs 213 Exploring NETCONF 220 Automating Using Network APIs 229 Using the Python requests Library 230 Using the Python ncclient Library 259 Using netmiko 284 Summary 289 8. Source Control with Git. 291 Use Cases for Source Control 291 Benefits of Source Control 292 Change Tracking 292 Accountability 292 Process and Workflow 293 Benefits of Source Control for Networking 293 Enter Git 294 Brief History of Git 294 Git Terminology 295 Overview of Git’s Architecture 296 Working with Git 297 Installing Git 297 Creating a Repository 297 Adding Files to a Repository 298 Committing Changes to a Repository 300 Changing and Committing Tracked Files 303 Unstaging Files 306 Excluding Files from a Repository 309 Viewing More Information About a Repository 313 Distilling Differences Between Versions of Files 317 Branching in Git 321 Creating a Branch 326 Checking Out a Branch 327 Merging and Deleting Branches 329 Collaborating with Git 334 Collaborating Between Multiple Systems Running Git 334 Collaborating Using Git-Based Online Services 351 Summary 355 x | Table of Contents 9. Automation Tools. 357 Reviewing Automation Tools 357 Using Ansible 359 Understanding How Ansible Works 360 Constructing an Inventory File 361 Executing an Ansible Playbook 368 Using Variable Files 373 Writing Ansible Playbooks for Network Automation 375 Using Third-Party Ansible Modules 393 Ansible Summary 396 Automating with Salt 396 Understanding the Salt Architecture 397 Getting Familiar with Salt 400 Managing Network Configurations with Salt 416 Executing Salt Functions Remotely 425 Diving into Salt’s Event-Driven Infrastructure 427 Diving into Salt a Bit Further 433 Salt Summary 436 Event-Driven Network Automation with StackStorm 436 StackStorm Concepts 437 StackStorm Architecture 439 Actions and Workflows 440 Sensors and Triggers 450 Rules 452 StackStorm Summary 455 Summary 455 10. Continuous Integration. 457 Important Prerequisites 459 Simple Is Better 459 People, Process, and Technology 460 Learn to Code 460 Introduction to Continuous Integration 460 Basics of Continuous Integration 461 Continuous Delivery 463 Test-Driven Development 464 Why Continuous Integration for Networking? 466 A Continuous Integration Pipeline for Networking 467 Peer Review 468 Build Automation 474 Test/Dev/Staging Environment 479 Deployment Tools 482 Table of Contents | xi Testing Tools and Test-Driven Network Automation 484 Summary 486 11. Building a Culture for Network Automation.