( Network == Server ) { Magic Happens }

Total Page:16

File Type:pdf, Size:1020Kb

( Network == Server ) { Magic Happens } If ( Network == Server ) { Magic happens } Attilla de Groot Sr. Systems Engineer, HCIE #3494 | Cumulus Networks 1 Cumulus Linux ArChiteCture Space Network Orchestration Automation Monitoring Routing User User Suite switchd Bridging VXLAN Third Party/Customer Applications Switch HAL ASIC Routing ARP Bridge Ethernet VxLAN Driver Tables Table Table Interfaces Linux Kernel CPU, RAM, Flash, etc. Switch Silicon Front Panel Ports 2 Network OrChestration Software deFining your network How to do Orchestration Linux DevOPS tools § AnsiBle, Puppet, Chef, Saltstack OrChestration § Infrastructure as code § Telemetry data § Continuous integration 3 Who recently made a network change? 4 Did anyone had a change without issues? 5 What is CI / CD Continuous Integration (CI) Continuous Deployment (CD) A system where all changes are Built on a CI system where changes automatically tested Before Being are automatically pushed to production pushed to production or seen By after tests past, often multiple times others per day Why aren’t you Not for doing this? everyone 6 InFrastruCture as Code Build tools Implementing CI/CD Build Tool? § The digital duct tape Common Tools § GitlaB § Jenkins § Travis CI § Atlassian BamBoo GitLaB 7 InFrastruCture as Code Pipeline Git change Successful push validation Infrastructure 1.1 Infrastructure Build tool pipeline 1.0 Infrastructure 1.0 Automated Validation Testing failed 8 Automated testing Testing inFrastruCture Traditional networking Cumulus Linux Testing environment Testing environment § Physical laB § Physical laB? § Virtualization environment? § Cumulus VX DevOPS integration? DevOPS integration! § Proprietary modules § Native modules § Vendor tools § Vagrant, VirtualBox, LiBvirt InfrastruCture validation InfrastruCture validation § Vendor APIs? § Home grown tools (Python?) § Stackstorm § Cumulus NetQ 9 Automated testing Cumulus VX Linux network virtualization Cumulus VX § Linux networking § Familiarizing Copying produCtion § VirtualBox, VMware, LiBvirt § Vagrant § Topology Generator 10 Automated testing Validation Linting tests Unit and System tests Code validation Validating the inFrastruCture § Test Yaml / Jinja § Unit tests are limited § Enfore a style § ComBining Unit and System tests § easy trouBleshooting Testing multiple verticals § Does my BGP session work? § Are routes Being learned? AND • Is my application redundant? • Does my application still work? 11 Automated testing Cumulus NetQ Design Provision Automate Validate TrouBleshoot Analyze Day 0 to day 1 Day 2 and on Cumulus Linux Cumulus NetQ Open Network Operating System Telemetry-Based Fabric validation application Open OS foundation of network Validate changes flexiBility & choice of apps Before & during production rollout Disaggregation of SW & HW Precise fault location alerts supply chain freedom for rapid proBlem resolution Speed of provisioning workloads Time-machine diagnosis & troubleshooting with automation at scale 12 Automated testing NetQ Agent NetQ Agent NetQ Agent SubsCribes to Linux Kernel Events § Interface State BGP mLAG VxLAN § MTU § Routes learned § Macs learned § (M)LAG state Linux Kernel Polls routing InFormation § OSPF, PIM, BGP, EVPN § NeighBor state § error conditions 13 Automated testing Fabric state testing over time NetQ Agent New Route Added OSPF NeighBor Change BGP mLAG VxLAN MAC Address Removed Linux Kernel 14 15 Automated testing Gitlab testing Test results returned Checks for updates GitLaB Pulls code update and runs tests git push Developer Build Server 16 Automated testing Gitlab testing stages: - staging Gitlab Pipeline - production staging: tags: - staging before_sCript: Stages, before_script, after_script - cd automation § Stages run in order on success stage: staging script: § *_script run at each stage - 'ansible-playbook deploy.yml' - sleep 25 Script deFines testing steps - netq check bgp - netq check mtu § Git repository is cloned on Build server - netq check vxlan § each script step is run in environment production: § If step fails, validation fails tags: - production § Simple Bash scripts can Be added before_sCript: - cd automation stage: production BeFore/After stages For setup / when: manual cleanup script: - 'ansible-playbook deploy.yml' § vagrant up / vagrant destroy -f - sleep 10 - netq check bgp 17 CI/CD Demo 18 Join the Linux Networking Revolution Linux 101 Linux Networking Linux ResourCe eBook Guides Center www.cumulusnetworks.com/lp/linux-networking-resources/ QUESTIONS? 19 Join the Cumulus Networks Community! Get involved in the conversation: § Join the community & share your Cumulus Linux tips and experiences: https://getsatisfaction.cumulusnetworks.com/cumulus § Chat with others on our Slack channel: https://cumulusnetworks.slack.com/ § Keep on learning: https://community.cumulusnetworks.com Download Cumulus VX to work on proofs of concept, test your automation and continue practicing: § https://cumulusnetworks.com/cumulus-vx/ Learn at your own pace with demos using our reference topology: § https://githuB.com/CumulusNetworks/cldemo-vagrant § https://cumulusnetworks.com/products/cumulus-in-the-cloud/ 20 Thank you! 21.
Recommended publications
  • A Study of the Effectiveness of Cloud Infrastructure
    A STUDY OF THE EFFECTIVENESS OF CLOUD INFRASTRUCTURE CONFIGURATION A Thesis Presented to the Faculty of California State Polytechnic University, Pomona In Partial Fulfillment Of the Requirements for the Degree Master of Science In Computer Science By Bonnie Ngu 2020 SIGNATURE PAGE THESIS: A STUDY OF THE EFFECTIVENESS OF CLOUD INFRASTRUCTURE CONFIGURATION AUTHOR: Bonnie Ngu DATE SUBMITTED: Spring 2020 Department of Computer Science Dr. Gilbert S. Young _______________________________________ Thesis Committee Chair Computer Science Yu Sun, Ph.D _______________________________________ Computer Science Dominick A. Atanasio _______________________________________ Professor Computer Science ii ACKNOWLEDGEMENTS First and foremost, I would like to thank my parents for blessing me with the opportunity to choose my own path. I would also like to thank Dr. Young for all the encouragement throughout my years at Cal Poly Pomona. It was through his excitement and passion for teaching that I found my passion in computer science. Dr. Sun and Professor Atanasio for taking the time to understand my thesis and providing valuable input. Lastly, I would like to thank my other half for always seeing the positive side of things and finding the silver lining. It has been an incredible chapter in my life, and I could not have done it without all the love and moral support from everyone. iii ABSTRACT As cloud providers continuously strive to strengthen their cloud solutions, companies, big and small, are lured by this appealing prospect. Cloud providers aim to take away the trouble of the brick and mortar of physical equipment. Utilizing the cloud can help companies increase efficiency and improve cash flow.
    [Show full text]
  • Lessons Learned from a Large-Scale Infrastructure As Code Project
    PowerShell Conference Europe 2019 Hannover, Germany June 4-7, 2019 Lessons learned from a large-scale infrastructure as code project Platinum Sponsor Mark Warneke After this Session - I am able to develop a mature “Infrastructure As Code” project from scratch using a Test-Driven development approach, avoiding common pitfalls and getting a heads up in necessary considerations, tools and best practices - I can build sophisticated Azure Release Pipelines that leverage advanced testing scenarios using Azure Resource Manager Templates, PowerShell tooling to support an advanced “Infrastructure As Code” project @MarkWarneke Agenda Introduction Architecture Demo @MarkWarneke By viewing cloud computing as a starting point for IT automation, companies may be able to have it all: scalability, agility, flexibility, efficiency, “ and cost savings. But that’s only possible by building up both automation and cloud capabilities. – McKinsey @MarkWarneke“ What is the challange? Speed Control Agility @MarkWarneke Why change? Servers Services @MarkWarneke Paradigm shift Enforce/Control Enable/Support Controlled & central responsibility Freedom & delegated responsibility @MarkWarneke What organization want Secure, predictable, and flexible Control service delivery and operations capability (end to end traceability). Faster business innovation through Innovation adoption of cloud services. Business agility and reduced time-to- Speed/Agility market through efficient DevOps teams. Costs Efficient use of public cloud scale. @MarkWarneke DevOps benefits based on research Source: 2018 State of DevOps Report DORA @MarkWarneke A Cloud Center of Excellence (CCoE) is a cross- functional team of people responsible for developing and managing the cloud strategy, “governance, and best practices that the rest of the organization can leverage to transform the business using the cloud.
    [Show full text]
  • Adoption, Support, and Challenges of Infrastructure-As-Code: Insights from Industry
    Adoption, Support, and Challenges of Infrastructure-as-Code: Insights from Industry Michele Guerriero,1 Martin Garriga,2 Damian A. Tamburri,3 Fabio Palomba4 1Politecnico di Milano, Italy 2Jheronimus Academy of Data Science & Tilburg University, The Netherlands 3Jheronimus Academy of Data Science & Eindhoven University of Technology, The Netherlands 4University of Zurich, Switzerland [email protected], [email protected], [email protected], palomba@ifi.uzh.ch Abstract—Infrastructure-as-code (IaC) is the DevOps tactic semi-structured interviews in as many companies to distill: (1) of managing and provisioning infrastructure through machine- how practitioners currently develop infrastructural code — that readable definition files, rather than physical hardware config- is, the best/bad practices experienced by the practitioners as uration or interactive configuration tools. From a maintenance and evolution perspective, the topic has piqued the interest of IaC blueprints grow in size and maintenance costs [8], (2) practitioners and academics alike, given the relative scarcity of what is the automatic support available — that is, pros and supporting patterns, best practices, tools, and software engineer- cons of using existing tools in practice, and (3) the challenges ing techniques. Using the data coming from 44 semi-structured reported by the practitioners via direct experience — that is, interviews in as many companies, in this paper we shed light the research and industrial avenues that practitioners perceive on the state of the practice in the adoption of IaC and the key software engineering challenges in the field. Particularly, as worthy of investigation in the near future. we investigate (i) how practitioners adopt and develop IaC, (ii) Data on the above points reveals a number of results as well which support is currently available, i.e., the typically used as valuable avenues for further work.
    [Show full text]
  • INFRASTRUCTURE AS CODE–FINAL REPORT John Klein, Phd and Douglas Reynolds December 2018
    INFRASTRUCTURE AS CODE–FINAL REPORT John Klein, PhD and Douglas Reynolds December 2018 1 Introduction This report concludes work on Research Project 6-18 518 Feasibility of Infrastructure as Code, summarizing the problem addressed by the research, the research solution approach, and results. 1.1 Background 1.1.1 What Is Infrastructure as Code? Infrastructure as code (IaC) is a set of practices that use “code (rather than manual commands) for setting up (virtual) machines and networks, installing packages, and configuring the environment for the application of interest” [3]. The infrastructure managed by this code includes both physical equipment (“bare metal”) and virtual machines, containers, and software-defined networks. This code should be developed and managed using the same processes as any other software; for example, it should be designed, tested, and stored in a version-controlled repository. Although information technology (IT) system operators have long employed automation through ad hoc scripting of tasks, IaC technology and practices emerged with the introduction of cloud computing, and particularly infrastructure-as-a-service (IaaS) technology. In an IaaS-based envi- ronment, all computation, storage, and network resources are virtualized and must be allocated and configured using application programming interfaces (APIs). While cloud service providers furnish management consoles that layer an interactive application on top of the APIs, it is not practical to use a management console to create a system with more than a couple of nodes. For example, creating a new virtual machine (VM) using the Amazon Web Services management console requires stepping through at least five web forms and filling in 25 or more fields.
    [Show full text]
  • Adopting 'Infrastructure As Code' Misha Zynovyev (GSI)
    Adopting 'Infrastructure as Code' Misha Zynovyev (GSI) adopting 'infrastructure as code' to run HEP applications Misha Zynovyev ([email protected]) Dennis Klein ([email protected]) Victor Penso ([email protected]) HEPiX Spring 2011 GSI, Darmstadt, 5th of May 2011 Adopting 'Infrastructure as Code' Misha Zynovyev (GSI) acknowledgements Dr. Peter Malzacher Bastian Neuburger Christopher Huhn Dr. Stefan Haller Helge Brust Oliver Wirth Adopting 'Infrastructure as Code' Misha Zynovyev (GSI) outline of the talk our mission statement current approaches to infrastructure management our approach to running HEP applications describing infrastructure with code our playgrounds results outlook Adopting 'Infrastructure as Code' Misha Zynovyev (GSI) clarification target audience: developers, system administrators and infrastructure providers you accept the benefits of virtualization presented concepts are transparent to users1 this is not about how to build the Infrastructure-as-a-Service (IaaS) clouds, but about efficient ways to use them 1nevertheless users can have as much control of details as they can cope with Adopting 'Infrastructure as Code' Misha Zynovyev (GSI) our mission statement we want to simplify deployment and operation of scientific computing applications and their infrastructure by describing everything in code furthermore, we need a mechanism which will allow us to use external resources provided by IaaS clouds in a transparent way Adopting 'Infrastructure as Code' Misha Zynovyev (GSI) deploying infrastructure manual (with OS tools, text-editor):
    [Show full text]
  • CV - Russell Dunphy [email protected] | 07849465404 |
    CV - Russell Dunphy [email protected] | 07849465404 | https://www.linkedin.com/in/rsslldnphy/ An experienced software developer, technical leader, and manager, with a love of functional ​ ​ ​ ​ ​ ​ ​ programming and coding as language. Came to programming by a somewhat unusual route: ​ ​ ​ a first class degree in Creative Writing. I see people, teams, code, and infrastructure as an interconnected ecosystem, and enjoy ​ ​ tending that ecosystem by sitting at the intersection of them all: doing people management, process optimisation, architecture, and coding as and when needed. I also enjoy developing long-term technical strategy, taking into account how this ecosystem fits into the wider business ​ ​ context, and finding sustainable, iterative ways to implement it. I’ve managed teams ranging in size from two to twelve, including developers, data scientists, scrum masters and testers. I’ve worked with a number of languages, most recently Clojure, ​ ​ Clojurescript, and Javascript/Typescript, and love learning more. I’m happy wrangling ​ ​ ​ ​ ​ infrastructure, doing backend work, or building frontend features. Most of all I like finding ways to make these different levels of the stack, and the people who work on them, form a cohesive whole. Employment Head of Engineering - HealthUnlocked Jun 2016 - Present ​ HealthUnlocked is social network for health, with around 3-4 million unique sessions per month. As Head of Engineering I am responsible for managing the tech team; for recruitment; for setting technical strategy for the organisation;
    [Show full text]
  • Devops: Introducing Infrastructure-‐As-‐Code
    DevOps: Introducing Infrastructure-as-Code Elisabetta Di Nitto, Damian A. Tamburri*, Michele Guerriero*, Matej Artac+, Tadej Borovsak+ *Politecnico di Milano +XLAB Corp. Roadmap o Session 1: DevOps In a Nutshell o Break! (5 mins) o Session 2: Infrastructure-as-code Explained through TOSCA o Session 3: Our proposal to connect Dev and Ops: the DICER tool Session 1 DevOps In a Nutshell Dev & Ops: the classical roles Dev focuses on o Developing features o Changing requirements o Releasing products Ops focuses on o Offering services o Providing guarantees and stability Dev vs. Ops “The system works correctly in the development machine but it does not on the operation machine” “10 days after successful deployment of last release, the system is overloaded” Who is guilty? Dev– team, or –Ops team? The problem: two disconnected groups • Dev works to apply releases • Ops resists to releases • Dev does not pay attention • Ops is aiming at to QoS guarantees guaranteering QoS • Changes are fundamental for the business! • QoS guarantees are needed too! DevOps • What is it: “Practices or tools that bridge the gap between development and operations” • Goal: Creates a collaborative mindset where a single team performs Dev and Ops àthe team must contain differentiated competences, background, etc. • Requires: • Culture management; • Automation tools; • Organisational as much as technical metrics • Continuous sharing artifacts, procedures, languages, approaches… DevOps Need 1: Process Alignment! - Unified Processes - Unified tooling Before DevOps Development
    [Show full text]
  • Infrastructure As Code (Iac) to Reduce Provisioning Time by 65%
    Infrastructure as Code (IaC) to Reduce Provisioning Time by 65% Credits Deepak Jayagopal Kanapathi Raja Sathya Narayanan Higher Elasticity And Scale of Infrastructure Has Made it Cumbersome to Manually Provision Even After The Adoption of Cloud Computing Legacy manual provisioning Cloud manual provisioning Evolution of of infrastructure infrastructure • Traditionally, Infrastructure provisioning has • Using cloud computing (Infrastructure as provisioning always been a manual process. a Service), an instant computing • Teams would rack and stack the servers and infrastructure could be provisioned and will manually configure them. managed over the internet. • Finally, they will install the application over • Cloud computing reduced the need for that hardware. high upfront capital expenditure as well • This used to be a slow process and there were as maintenance costs. a lot of accuracy and consistency issues. Infrastructure as Code (IaC) Infrastructure as Code IaC allows DevOps IaC takes cloud (IaC) is the process of engineers to view computing to its provisioning and complex maximum potential by managing the entire infrastructure in a automating the infrastructure through a codified way. manual, error-prone series of software. provisioning tasks. Cloud infrastructure solved many issues pertaining to the legacy provisioning methods. However, there are still issues persisting with manual provisioning of cloud infrastructure. Confidential & Restricted 2 Major Challenges with Manual Provisioning of Cloud Infrastructure DSPs across the globe have a Time consuming in Higher cost great necessity to adopt a provisioning infrastructure strong cloud strategy to deliver digital services across In the case of huge infrastructures, the time Cost is calculated per hour by taken to manual cloud provisioning gets cloud vendors.
    [Show full text]
  • Co-Evolution of Infrastructure and Source Code - an Empirical Study
    Co-evolution of Infrastructure and Source Code - An Empirical Study Yujuan Jiang, Bram Adams MCIS lab Polytechnique Montreal, Canada Email: yujuan.jiang,bram.adams @polymtl.ca { } Abstract—Infrastructure-as-code automates the process of have a high coupling with source code and test files, which configuring and setting up the environment (e.g., servers, VMs means that developers and testers need to perform a certain and databases) in which a software system will be tested and/or effort to maintain the build system files as the code and tests deployed, through textual specification files in a language like evolve. Based on these findings, we conjecture that IaC could Puppet or Chef. Since the environment is instantiated auto- run similar risks and generate similar maintenance overhead matically by the infrastructure languages’ tools, no manual as regular build scripts. intervention is necessary apart from maintaining the infrastruc- ture specification files. The amount of work involved with such In order to validate this conjecture, we perform an empir- maintenance, as well as the size and complexity of infrastructure ical case study on 265 OpenStack projects. OpenStack is an specification files, have not yet been studied empirically. Through ecosystem of projects implementing a cloud platform, which an empirical study of the version control system of 265 OpenStack requires substantial IaC to support deployment and tests on projects, we find that infrastructure files are large and churn frequently, which could indicate a potential of introducing bugs. virtual machines. The study replicates the analysis of McIntosh Furthermore, we found that the infrastructure code files are et al.
    [Show full text]
  • Infrastructure As Code Handbook
    THE INFRASTRUCTURE AS CODE HANDBOOK Best practices, tools, and AWS CloudFormation scripts you can use to implement IaC for your organization www.thorntech.com Copyright © 2018 | THORN TECHNOLOGIES All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the publisher. THE INFRASTRUCTURE AS CODE HANDBOOK Contents 1 FIVE REASONS WHY YOU SHOULD IMPLEMENT INFRASTRUCTURE AS CODE NOW 2 SIX BEST PRACTICES TO GET THE MOST OUT OF IAC 3 FIFTEEN IAC TOOLS YOU CAN USE TO AUTOMATE YOUR DEPLOYMENTS 4 WHAT IS AWS CLOUDFORMATION AND HOW CAN IT HELP YOUR IAC EFFORTS? 5 HOW AWS CLOUDFORMATION WORKS AND HOW TO CREATE A VIRTUAL PRIVATE CLOUD WITH IT 6 HOW TO INCORPORATE S3, EC2, AND IAM IN A CLOUDFORMATION TEMPLATE 7 HOW TO CREATE A REDSHIFT STACK WITH AWS CLOUDFORMATION 3 www.thorntech.com [email protected] THE INFRASTRUCTURE AS CODE HANDBOOK 5 reasons why you should implement infrastructure as code now INFRASTRUCTURE AS CODE CAN HELP AVOID CLOUD DEPLOYMENT INCONSISTENCIES, INCREASE DEVELOPER PRODUCTIVITY, AND LOWER COSTS There’s no doubt that cloud computing has had a major impact on how companies build, scale, and maintain technology products. The ability to click a few buttons to provision servers, databases, and other infrastructure has led to an increase in developer productivity we’ve never seen before. While it’s easy to spin up simple cloud architectures, mistakes can easily be made provisioning complex ones. Human error will always be present, especially when you can launch cloud infrastructure by clicking buttons on a web app.
    [Show full text]
  • Microsoft Azure: Infrastructure As Code
    Microsoft Azure: Infrastructure as Code WorkshopPLUS Duration: 3 Days Focus Area: Change and Configuration Management Difficulty: 300 - Advanced Overview Key Takeaways Get hands-on with Infrastructure as Code by learning how Course Material to author and submit code from common tools into Azure • Azure Templates Resource Manager (ARM). In this course you can choose learning paths for integrating with Microsoft 1st party and • Visual Studio Code 3rd party open source including: Azure Templates, Hands-on Labs Terraform, Ansible, Azure Automation, and Desired State • Most of the concepts covered above will be supported Configuration. You’ll walk away understanding basic and by hands-on labs and demos advanced techniques for programmatically instructing Azure how to build-out and update your environments. • Utilize tools that integrate with Azure Objectives Agenda After completing this training, students will be able to: Day 1 • Understand what infrastructure as code means • Introduction to Infrastructure as Code • Learn about the different tools available to manage • Tools code lifecycle • Overview of Azure • How to manipulate & reference code Day 2 • Optional Module(s) • Understand the structure of an Azure Template Day 3 • Set resource deployment order • Optional Module(s) • Onboard Azure Automation & DSC • Q&A, as time permits • Learn to leverage & integrate third-party, open source software to develop the Azure environment Plan for 3 full days. Early departure on any day is not • Describe Ansible fundamentals, dynamic inventory and recommended. deploy Azure resources • Learn Terraform best practices for deploying Azure resources • Integrate Terraform with continuous integration / continuous delivery 2019 © Microsoft Corporation. All rights reserved. This data sheet is for informational purposes only.
    [Show full text]
  • HASHICORP TERRAFORM and RED HAT ANSIBLE AUTOMATION Infrastructure As Code Automation
    HASHICORP TERRAFORM AND RED HAT ANSIBLE AUTOMATION Infrastructure as code automation OVERVIEW INTRODUCTION As organizations modernize their application delivery process and adopt new tools to make them more efficient, infrastructure as code (IaC) has become a critical practice. IaC is an umbrella term, usually referring to a broad range of tools, operational frameworks, and a rich ecosystem. In this piece, we describe what is meant by infrastructure as code and how HashiCorp Terraform and Red Hat® Ansible® Automation can be used as a foundation for an IaC initiative. WHAT IS INFRASTRUCTURE AS CODE? In the last four decades, most approaches to infrastructure management have relied on graphical and command-line user interfaces that human operators have used to initialize or modify system configurations. These approaches are prone to human error. The process carried out by an operator might be documented by a checklist or a memorized routine without enforced oversight or iterative versioning. In contrast, the IaC approach promotes formalized, standardized, and automated operational processes — and dictates that these operational processes are documented as configuration files or programming code. By treating infrastructure as code, IT organizations can automate management tasks while using the best practices of software development, including code review and version control. This approach mitigates management complexity by breaking down a task into smaller, more manageable processes, controlling the execution of code, and effortlessly maintaining up-to-date documentation. The IaC approach also reduces operational risks by allowing multiple subject matter experts to peer review the code and by saving all the previous revisions of a codified infrastructure, enabling previous versions to be restored in case of mistakes.
    [Show full text]