Cumulus Certified Open Network Professional Study Guide CUMULUS NETWORKS / CCONP STUDY GUIDE

CUMULUS NETWORKS / CCONP STUDY GUIDE

Purpose

To help certification candidates organize their training and study plan, matching directly to the exam blueprint with information and resources to augment Cumulus Networks training. Some general networking exposure and knowledge is assumed, but links are always provided for additional research at your own consumption pace.

Some general images of packet types or other reference information were included from web sources such as Wikipedia and vendor web sites for quick reference.

Organization

This study guide was organized and generated directly from the exam study guide blueprint with modifications and additions deemed appropriate.

https://education.cumulusnetworks.com/getting-started-materials/287534

Creation references

The document was created primarily using the Cumulus Linux 3.7 User Guide, Cumulus NetQ 1.4 User Guide (commands validated in version 2.1), validated design documents, Cumulus provided free training resources, and boot camp documentation. Additional information was added from prior knowledge and research.

·· Knowledge Base Home — https://support.cumulusnetworks.com/hc/en-us

·· Cumulus Education Home — https://education.cumulusnetworks.com/

·· Cumulus Linux User Guide — https://docs.cumulusnetworks.com/display/DOCS

·· Cumulus NetQ User Guide — https://docs.cumulusnetworks.com/display/NETQ

·· Cumulus Validated Design Guides — https://cumulusnetworks.com/learn/resources/installation-guides

·· Cumulus Product Collateral — https://cumulusnetworks.com/learn/resources/cumulus-linux

·· Cumulus Data Center Networking CheatSheets — https://cumulusnetworks.com/learn/ resources/cheatsheetsf

Document formatting

Code, configuration, and examples The document contains a lot of examples of commands and output. Some commands output may be slightly formatted to fit inside this document and large tables may be reduced to the number of rows required for the clarity of information.

Code or configuration examples will be shown in grey background text container in Courier New 9pt font. After the example will be a blank line formatted with minimal spacing. Crowded examples may include bold on commands for emphasis.

Files, directories, paths, and commands without output Files, directories, paths and singular command references lacking output will be displayed in italics, /etc/cumulus/acl/policy.conf with the font and text size it is included in.

Important headings and focused text will be highlighted in bold.

Commands and syntax Commands for Cumulus Linux example will show the command and assume a not shown net commit command is run to activate the changes before the verification. Some command output was restructured to fit into the document without changing the content itself, such as table layout innet show interface command.

Command syntax that is showing options within commands and not the command itself with output will be written with the following syntax:

·· Required variable information enclosed in greater than and less than symbols “”

··

·· Optional items or sections will be enclosed in brackets, “[y]”

·· [optional_section]

·· Required items with fixed choice selection will be enclosed in parentheses, “(z)”

·· (choice1|choice2)

·· Some choices may be omitted for brevity

NCLU & NetQ When the same information is available to be determined for show commands or troubleshooting, both examples were attempted to be included. NCLU was prioritized since NCLU is included by default compared to the optional NetQ added to provide troubleshooting and validation at a higher scale. Some capability is currently not in parity and can only be found via either NCLU or NetQ.

Contents

Purpose 1

Organization 1

Creation references 1

Document formatting 1 Code, configuration, and examples 1 Files, directories, paths, and commands without output 2 Commands and Syntax 2 NCLU & NetQ 2

Exam information and training resources 11 Overview 11 Free resources and training 12 Self-paced training 12 Linux Networking fundamentals 12 Cumulus core 12

Instructor led training 13 Boot camp 13 Boot camp XL 13 Schedule exam 13

Switching fundamentals 13 Describe & switching concepts 13 Frame switching 13 Frame flooding 13 MAC address table 14 MAC learning and aging 15 Interpret frame format 15

Configure, verify, and troubleshoot inter-VLAN bridging 16 VLAN trunking 16

Describe Linux bridges concepts 16 Describe VLAN aware bridge 16 Describe traditional bridging concepts 17 Traditional | VLAN-aware bridging comparison 17

Describe and verify ARP and neighbor discovery 18 ARP overview 18 ARP changes in Cumulus Linux 18 ARP verification & manipulation 19 Neighbor discovery 20

Configure, verify, and troubleshoot STP protocols 21 Describe supported STP modes and interop techniques 21 Configuration 21 Storm control 23 Verification 23 Troubleshooting 24

Configure and verify layer 2 protocols 25 Multi-chassis Link Aggregation (MLAG) 25 MLAG with STP 26 MLAG log file 26 Describe ethernet bridging fundamentals 27

Describe & configure connectivity to the host 27 Describe common host attachment modes 27 Describe the purpose of Multi-chassis Link Aggregation (MLAG) 28

Routing fundamentals 28 Describe BGP and how it is used 28 Border Gateway Protocol (BGP) overview 28

Describe the differences between AS placements EBGP vs. IBGP 30 iBGP 30 eBGP 30

Describe how OSPF is used and LSA types 31 Open Shortest Path First (OSPF) overview 31 OSPF as a DC underlay 32 OSPF area placement 32 OSPF stub areas 32

Describe the components of FHRP 33 Switched virtual interface 33 Virtual Router Redundancy (VRR) 33 Virtual Router Redundancy Protocol (VRRP) 34 Anycast gateway 36

Describe the components of a routing table 36 Describe Equal Cost Multipath (ECMP) routing 36 Describe hashing 37 Define equal cost 37 Comparing different sources like OSPF and BGP routes 37

Describe how a routing table is populated by different routing information sources 38 Compare and contrast static routing and dynamic routing 40 Compare and contrast different routing protocols 40 BGP 40 OSPF 40 Routing Information Protocol (RIP) 40 Intermediate System to Intermediate System (IS-IS) 41 Enhanced Interior Gateway Routing Protocol (EIGRP) 43 Quick comparison chart 44

Describe IPv4 and IPv6 addressing fundamentals 44 IPv4 addressing overview 44 IPv6 overview 45

Configure, verify, and troubleshoot IPv4 and IPv6 static routing 47 IPv4 static route configuration 47 IPv4 static route verification 47 IPv6 static route configuration 47 IPv6 static route verification 47

Describe the Linux theory on VRF 48 Virtual Routing and Forwarding (VRF) overview 48 Describe MGMT VRF theory 48 Configure VRF 49 Configure management VRF 49 VRF verification 49 VRF route table 50

mcast (no PIM) 51 Describe IGMP functionalities 51

Linux concepts 51 Describe the basics of GRUB 51 Display how to boot a switch, recover a password, and manually boot 52 Restart ONIE installer 52 Uninstall all images & remove configuration 52

Boot into rescue mode 52 Password recovery 53

Installing software and package management (.deb, source...etc.) — high-level concepts 54 Understand how to use a change log 54

Display how to add and remove users, set permissions on files, password 55 Add and remove users 55 Set password 55 Set file permissions 56

Describe the benefits and differences between password login and keybased 57 Describe the difference between Userspace and Kernel 57 Configure systemd service architecture 58 Display starting, enabling, disabling a service 58

BASH overview and purpose 59 Stdin/out/err, utilities, pipes and redirection 59 Pipes 59 Display how to change directories 60 Display how to create files 60 Display how to use sudo 61 Display how to use grep 61

Describe file system structure and where files are located 62 Cumulus Linux Network configuration files 63 Additional commonly used files 64

Dynamic Host Configuration Protocol (DHCP) 64

Overlay routing concepts 65 Describe and configure a VXLAN 65 VXLAN overview 65 VXLAN configuration 66

Describe the difference between asymmetric and symmetric routing 66 Asymmetric routing 66 Symmetric routing 67

Describe the basics of EVPN, a BGP EVPN control plane, and the different route types 69 Ethernet Virtual Private Network (EVPN) 69 BGP EVPN control plane 69 EVPN route types 70

Core Cumulus concepts 72 Describe awareness & interaction between NCLU & ifupdown2 72 Configure interfaces 73 Create a topology file and verify cabling with PTM 75 PTM overview 75 Basic DOT example 76 PTM templates 76

Configure, describe, and troubleshoot BGP unnumbered operation 77 BGP unnumbered overview 77 BGP unnumbered configuration 78 BGP unnumbered troubleshooting 78 Link-local addresses validation 84 Describe how to manage FRR 85

Describe NCLU and display how to leverage help, add/remove config 87 NCLU overview 87 NCLU help 88 NCLU built in examples 90

Describe hardware abstraction 91 Switchd 91 Netlink interaction with Kernel 92

Describe purpose of ONIE 92 Describe how a system boots, how to install an OS 92 Describe ZTP 95 Cumulus Linux ZTP options 95 ZTP best practices (can expand each to tier 3 and apply details to each) 95

Control plane protection ACL & SPAN with cl-acltool 95 Netfilter ACLs, cl-acltool, iptables, & ebtables overview 95 Netfilter tables 96 Netfilter chains 97 Netfilter rules 98 ACL rule assignment placement 98 Describe control plane protection with cl-acltool 98 Configure SPANs with cl-acltool 100 SPAN overview 100 Limitations for SPAN | ERSPAN with Cumulus Linux 101 Configuration steps 101 Selective SPAN 104

Layer 2 ebtables ACL example 104 SPAN with Tomahawk Based Switches 104 Other references 105

IPv6 and services 105 IPv6 105 AAA 105 RADIUS package install 105 RADIUS client configuration with local Fallback 106 Enabling login without local accounts 107 RADIUS client configuration verification 107 Network Time Protocol (NTP) 108 NTP with DHCP 109 NTP via NCLU 109 SNMP historical overview & components 110 Cumulus custom MIBs 111 SNMP configuration with NCLU 111 DHCP relay overview 112 DHCP relay IPV4 configuration with NCLU 112 DHCP relay IPv6 configuration 113 DHCP relay troubleshooting 114 DHCP relay RFC 3527 link selection sub-option 115

Design concepts 115 Describe Clos design 115 Describe various modern architecture designs 115 Traditional spanning tree — single attached 116 MLAG 116 L3 single-attached hosts 117 Redistributed Neighbor vs. Routing on Host (ROH) 117 Routing on the Virtual Machine (VM) 118 Virtual router 119 Routing on the Host (ROTH) BGP advertisement best practices 119 Anycast with manual redistribution 120 LNV with MLAG 120 Centralized VXLAN routing with bridging using EVPN design reference 121 Distributed VXLAN routing with asymmetric model design reference 121 Distributed VXLAN routing with symmetric model design reference 122

Describe service leafs 122 Describe Equal Cost Multipath (ECMP) 122 ECMP overview 122 ECMP hashing 122 ECMP resilient hashing 123 ECMP resilient hashing configuration 124

Describe oversubscription ratios 124 Describe port density, sizing of the DC 124 High performance leafs (nonblocking) 126

Troubleshooting 127 Describe basic troubleshooting techniques 127 Basic steps 127 Isolate the problem 127 Implementing a fix 128 Verifying the fix resolves the problem 128

Validate layer 1 via NetQ & NCLU methods 129 Verify link state 129 Verify counters 132 Verify bonding 133

Validate layer 2 via NetQ & NCLU methods 134 Verify spanning tree 134 Verify VLANs 135

Validate layer 3 via NetQ & NCLU methods 137 IP addressing 137 Route peering 138 Route table 141 EVPN 143

NetQ path trace 146 Linux system validation 147 Display & validate CPU utilization 147 Display & validate memory utilization 148 Display & validate disc utilization 149 Display & validate services 150

System environmental validation 151 Display & validate temperature 151 Display & validate fan speed 151 Display & validate Power Supply (PSU) 152

Automation 152 Identify potential automation templates 152 Describe the principles of automation 154 Describe a library/module 154 Describe groupings 154 Describe push vs. pull & agent vs. agentless 155 Push vs. pull 155 Agent vs. agentless 155

Describe idempotentcy 155 Name major automation vendors 155 Ansible 155 Puppet 156 Chef 156 SaltStack 156 CFEngine 156 Automation vendor table summary 156

Articulate Linux automation strategy (push file restart –> service) 157 Enable and use the NCLU API 157 Enabling API 157 NCLU API usage examples 158

Exam information and training resources

Overview The objective of Cumulus Networks training and exam content is to have the candidate walk away with Cumulus Linux proficiency and general Linux networking proficiency. The certification was created to fill gaps in existing training that either do not cover Linux or maintain heavy focus on proprietary vendors and information. Cumulus recommends a study, prepare, and test strategy when pursuing the certification.

This document was designed specifically from the exam blueprint (additional details added), so it will familiarize you with the general sections and outline of the exam which follows.

·· Switching Fundamentals

·· Routing Fundamentals

·· Linux Concepts

·· Overlay Routing Concepts

·· Core Cumulus Concepts

·· Design Architecture Concepts

·· Troubleshooting

The exam is ~90 questions, 2 hours, and delivered online at your own site and convenience. Upon passing, you will receive a number based on the year and order of passage, for example 2019::2. The certification is valid for 3 years and successfully recertifying will keep your existing number.

FIGURE 1

Free resources and training Refer to https://education.cumulusnetworks.com/getting-started-materials for the program overview, where to begin tips, and the exam blueprint.

Cumulus Networks offers a Linux 101 ebook and other free curriculum resources to build a base of knowledge.

https://education.cumulusnetworks.com/linux-101-ebook-educational-resources

They also provide free how-to videos covering open networking basics, Cisco configuration comparisons, NCLU, and Cumulus tutorials.

https://education.cumulusnetworks.com/series/how-to-videos

Self-paced training Linux networking fundamentals

https://education.cumulusnetworks.com/series/linux-fundamentals

Linux Networking Fundamentals training is 3.5 hours of training covering 9 areas of essential knowledge design for those new to Linux or networking.

·· Linux Concepts

·· IP Addressing

·· Routing Fundamentals

·· BGP Fundamentals

·· OSPF Fundamentals

·· Linux Routing Proficiency

·· First Hop Redundancy Cumulus core

https://education.cumulusnetworks.com/series/cumulus-core

4.5 hours of training focusing on operational knowledge for those new to cumulus and teaching them everything they need to be dangerous. The course is split into 5 modules.

·· Architecture

·· Configuration

·· Routing

·· Network Services and Security

·· Troubleshooting

Instructor led training Boot camp

https://cumulusnetworks.com/support/networking-training/

Cumulus Networks offers online boot camps with open enrollment, and lasting 12 hours equally spread over 3 days. You will by lead through practical exercise and hands on labs to strengthen your knowledge and understanding of the topics.

·· Module 1: Overview of Cumulus Linux

·· Module 2: Initial Setup

·· Module 3: Network Command Line Utility (NCLU)

·· Module 4: Interface Configuration

·· Module 5: Multi-chassis Link Aggregation (MLAG)

·· Module 6: Configuring Routing Protocols

·· Module 7: Data Center Architecture

·· Module 8: VXLAN EVPN

·· Module 9: Data Center Network Services

·· Module 10: Automation

·· Module 11: Switch Health and Monitoring

·· Module 12: Image Management Boot camp XL

On site dedicated to single company or organized group up to 16 people over 2 days for covering 16 hours. This option uses the same modules, but provides more depth and access to in person live responses from the instructor, and is great for and organization with a large team.

Schedule exam

https://education.cumulusnetworks.com/certification-exam-registration

Switching fundamentals

Describe & switching concepts Frame switching

Frames are the final layer of encapsulation before transmission over physical medium. Ethernet is an example of a network technology using frames for communication. Frame switching is the method by which the frames are transferred between hosts by a central switching device.

Frame flooding

Frame flooding is a method of frame switching to handle unknown destinations. The frame with an unknown destination is flooded to all ports except for the port the frame was received on.

MAC address table

A media access control (MAC) address table, sometimes referred to as a Content Addressable Memory (CAM) table, is used on Ethernet switches to determine how to forward frames in a local area network (LAN). This table stores both static and dynamic MAC addresses for forwarding of layer 2 frames. MAC addresses are 48 bits in length and usually displayed as 12 hexadecimal characters separated by colons or dashes.

cu mulus@leaf01:m g mt-vrf:~$ net show bridge macs VLAN Master Interface MAC TunnelDest State Flags LastSeen ------13 bridge bond01 44:38:39:00:08:01 00:00:21 13 bridge bond01 46:38:39:00:08:01 00:00:26 13 bridge bond01 46:38:39:00:08:02 00:00:25 13 bridge bridge 44:38:39:00:02:05 permanent 02:48:12 13 bridge bridge 44:39:39:ff:00:13 permanent 02:48:12 13 bridge peerlink 44:38:39:00:03:05 static 02:48:05 13 bridge vni13 44:38:39:00:0a:01 offload 02:47:26 13 bridge vni13 44:38:39:00:04:05 static offload 02:48:06 13 bridge vni13 44:38:39:00:05:05 static 02:48:06 13 bridge vni13 46:38:39:00:0a:01 offload 02:47:27 13 bridge vni13 46:38:39:00:0a:02 offload 02:47:27 24 bridge bond02 44:38:39:00:09:01 00:00:20 24 bridge bond02 46:38:39:00:09:01 00:00:26 24 bridge bond02 46:38:39:00:09:02 00:00:25 24 bridge bridge 44:38:39:00:02:05 permanent 02:48:12 24 bridge bridge 44:39:39:ff:00:24 permanent 02:48:12 24 bridge peerlink 44:38:39:00:03:05 static 02:48:05 24 bridge vni24 44:38:39:00:0b:01 offload 02:47:26 24 bridge vni24 44:38:39:00:04:05 static offload 02:48:06 24 bridge vni24 44:38:39:00:05:05 static 02:48:06

cu mulus@leaf01:m g mt-vrf:~$ netq sh macs Matching mac records: Origin MAC Address VLAN Hostname Egress Port Remote LastChanged ------no 44:38:39:00:02:05 13 leaf04 vni13:10.0.0.112 yes 2h:52m:34s no 44:38:39:00:02:05 13 leaf03 vni13:10.0.0.112 yes 2h:52m:33s no 44:38:39:00:02:05 24 leaf04 vni24:10.0.0.112 yes 2h:52m:33s no 44:38:39:00:09:01 24 leaf03 vni24:10.0.0.112 yes 2h:51m:53s no 44:38:39:00:09:01 24 leaf04 vni24:10.0.0.112 yes 2h:51m:53s no 46:38:39:00:0b:02 24 leaf01 vni24:10.0.0.134 yes 2h:51m:55s yes 44:38:39:00:02:05 13 leaf01 bridge no 2h:52m:38s

yes 44:38:39:00:02:05 13 leaf02 peerlink:leaf01 no 2h:52m:36s yes 44:38:39:00:02:05 24 leaf01 bridge no 2h:52m:37s yes 44:38:39:00:02:05 24 leaf02 peerlink:leaf01 no 2h:52m:36s

MAC learning and aging

Traditionally, MACs are learned from the source MAC in the frame ingress to a port, so that future frames with that destination MAC can be switched directly to that port rather than flooded.

Instead of keeping the learned MAC address permanently and potentially running out of storage space, the learned addresses are aged out after a certain time without being seen. By default, Cumulus Linux stores MAC addresses in the Ethernet switching table for 1800 seconds (30 minutes). This timer can be changed via NCLU.

The bridge-ageing option is in the NCLU blacklist, as it’s not frequently used. To configure this setting, you need to remove the bridge-ageing keyword from the ifupdown_blacklist in /etc/netd.conf. Restart the netd service after you edit the file. After restarting the service you can change the setting using NCLU.

cu mulus@switch:~$ net add bridge bridge ageing 600 cu mulus@switch:~$ net pending cu mulus@switch:~$ net com mit

Interpret frame format

https://en.wikipedia.org/wiki/Ethernet_frame

Ethernet Frames are comprised of a header, payload, and frame check sequence preceded by a preamble and start frame delimiter, and followed by an end of frame and interpacket gap.

FIGURE 2

The Ethernet frame can be encapsulated in Ethernet or another protocol to accomplish overlay functions. For example Virtual Extensible LAN (VXLAN) technology attempts to address scalability issues associated with hyper scale computing deployments. Its goal is to provide layer 2 connectivity through a layer 3 infrastructure avoiding the pitfalls of traditional layer 2 network spans.

FIGURE 3

Configure, verify, and troubleshoot inter-VLAN bridging VLAN trunking

https://docs.cumulusnetworks.com/display/DOCS/VLAN+Tagging

VLAN Trunking allows traffic assigned to multiple VLANs to transit a single link. VLANs are tagged for unique identification with a 4 byte 802.1q header.

FIGURE 4

Describe Linux bridges concepts Describe VLAN aware bridge

https://docs.cumulusnetworks.com/display/DOCS/VLAN-aware+Bridge+Mode

The VLAN-aware mode in Cumulus Linux implements a configuration model for large-scale L2 environments, with one single instance of Spanning Tree. Each physical bridge member port is configured with the list of allowed VLANs as well as its port VLAN ID. MAC address learning, filtering and forwarding are VLAN-aware. This significantly reduces the configuration size, and eliminates the large overhead of managing the port/VLAN instances as subinterfaces, replacing them with lightweight VLAN bitmaps and state updates.

The reasons to use VLAN-aware mode for bridges are:

·· Scale: The new VLAN-aware mode can support 2000 concurrent VLANs while the traditional mode supports only 200 concurrent VLANs

·· Simplicity: VLAN-aware mode has a simpler configuration

FIGURE 5

Cumulus Networks recommends using VLAN-aware mode bridges, rather than traditional mode bridges. The bridge driver in Cumulus Linux is capable of VLAN filtering, which allows for configurations that are similar to incumbent network devices. You can configure both VLAN-aware and traditional mode bridges on the same network in Cumulus Linux; however you should not have more than one VLAN-aware bridge on a given switch.

Describe traditional bridging concepts

https://docs.cumulusnetworks.com/display/DOCS/Traditional+Bridge+Mode

The only reasons to use traditional mode are:

·· Familiarity with traditional Linux syntax.

·· VXLAN support prior to CL 3.1: Starting with Cumulus Linux 3.1, VXLAN is supported by VLAN-aware mode bridges. For VXLAN support on earlier releases, use traditional mode.

·· PVSTP+ interoperability: The traditional mode currently runs an instance of spanning tree per bridge. The VLAN-aware STP mode is compatible with other types of spanning tree but only runs single instance MST. To achieve Per-VLAN STP/RSTP the traditional bridge mode must be used.

Traditional | VLAN-aware bridging comparison

https://support.cumulusnetworks.com/hc/en-us/articles/204909397

The following behaviors apply to both modes:

·· Network interfaces are configured under/etc/network/interfaces

·· Both modes support Spanning Tree Protocol

·· Interfaces configured with both modes can be managed withifupdown commands (ifup , ifdown )

Describe and verify ARP and neighbor discovery ARP overview

Address Resolution Protocol (ARP), defined in RFC 826 is a communication protocol used for discovering the link layer (MAC) address, associated with a given network layer (IP) address.

The Cumulus Linux ARP implementation differs from standard Debian Linux ARP behavior in a few ways because Cumulus Linux is an operating system for routers/switches rather than servers.

ARP changes in Cumulus Linux

Parameter Type Debian Cumulus

arp_accept BOOL 0 — Don’t create new entries in the ARP table.

arp_announce INT 0 — (Default) Use 2 — Always use the best local address for this target. In this any local address, mode we ignore the source address in the IP packet and configured on try to select local address that we prefer for talks with the any interface. target host. Such local address is selected by looking for primary IP addresses on all our subnets on the outgoing interface that include the target IP address. If no suitable local address is found we select the first local address we have on the outgoing interface or on all other interfaces, with the hope we will receive reply for our request and even sometimes no matter the source IP address we announce.

arp_filter BOOL 0 — (Default) The kernel can respond to ARP requests with addresses from other interfaces. This may seem wrong but it usually makes sense, because it increases the chance of successful communication. IP addresses are owned by the complete host on Linux, not by particular interfaces. Only for more complex setups like load- balancing, does this behavior cause problems.

arp_ignore INT 0 — (Default) 1 — Reply only if the target IP address is local address Reply for any local configured on the incoming interface. target IP address, configured on any interface.

arp_notify BOOL 0—(Default) 1 — Generate gratuitous arp requests when device is Do nothing. brought up or hardware address changes.

ARP verification & manipulation

https://docs.cumulusnetworks.com/display/DOCS/Network+Troubleshooting#NetworkTroubleshooting- ManipulatetheSystemARPCache

Display the ARP cache:

cu mulus@leaf01:m g mt-vrf:~$ arp -a ? (169.254.0.1) at 44:38:39:00:06:01 [ether] PERM on swp51 ? (10.1.3.12) at 44:38:39:00:03:05 [ether] PERM on vlan13 ? (192.168.0.254) at 44:38:39:00:01:01 [ether] on eth0 ? (10.2.4.12) at 44:38:39:00:03:05 [ether] PERM on vlan24 ? (169.254.0.1) at 44:38:39:00:07:01 [ether] PERM on swp52 ? (10.2.4.102) at 44:38:39:00:09:01 [ether] on vlan24 ? (169.254.1.2) at 44:38:39:00:03:03 [ether] on peerlink.4094 ? (10.1.3.101) at 44:38:39:00:08:01 [ether] on vlan13

Delete an ARP cache entry:

cu mulus@switch:~$ ar p -d 11.0.2.2 cu mulus@switch:~$ arp -a ? (11.0.2.2) at on swp3 ? (11.0.3.2) at 00:02:00:00:00:01 [ether] on swp4 ? (11.0.0.2) at 44:38:39:00:01:c1 [ether] on swp1

Add a static ARP cache entry:

cu mulus@switch:~$ ar p -s 11.0.2.2 00:02:00:00:00:10 cu mulus@switch:~$ arp -a ? (11.0.2.2) at 00:02:00:00:00:10 [ether] PERM on swp3 ? (11.0.3.2) at 00:02:00:00:00:01 [ether] on swp4 ? (11.0.0.2) at 44:38:39:00:01:c1 [ether] on swp1

To keep neighbors in the reachable state, Cumulus Linux includes a background process (/usr/bin/neighmgrd) that tracks neighbors that move into a stale, delay or probe state, and attempts to refresh their state ahead of any removal from the Linux kernel, and thus before it would be removed from the hardware forwarding.

cu mulus@leaf01:m g mt-vrf:~$ netq show services neighmgrd Matching services records: Hostname Service PID VRF Enabled Active Monitored Status Uptime LastChanged ------leaf01 neighmgrd 764 default yes yes no ok 7h:23m:3s 7h:21m:40s leaf02 neighmgrd 764 default yes yes no ok 7h:22m:44s 7h:21m:37s leaf03 neighmgrd 765 default yes yes no ok 7h:23m:14s 7h:21m:42s leaf04 neighmgrd 761 default yes yes no ok 7h:23m:7s 7h:21m:33s spine01 neighmgrd 759 default yes yes no ok 7h:22m:43s 7h:21m:36s spine02 neighmgrd 761 default yes yes no ok 7h:23m:17s 7h:21m:34s

Neighbor discovery

https://en.wikipedia.org/wiki/Neighbor_Discovery_Protocol

Hosts and routers use ND in IPv6 to determine link-layer addresses, and routers attached.

The Neighbor Discovery Protocol (NDP, ND) is a protocol in the Internet protocol suite used with IPv6. It operates at the link layer of the Internet model (RFC 1122), and is responsible for gathering information required for internet communication, including configuration of local connections, domain name servers, and gateways used to communicate with more distant systems.

The protocol defines five ICMPv6 packet types for IPv6 functions similar to the ARP and ICMP Router Discovery and Router Redirect protocols for IPv4, but provides improvements over its IPv4 counterparts (RFC 4861, section 3.1).

·· Router Solicitation (Type 133)

·· Hosts inquire with Router Solicitation messages to locate routers on an attached link. Routers which forward packets not addressed to them generate Router Advertisements immediately upon receipt of this message rather than at their next scheduled time.

·· Router Advertisement (Type 134)

·· Routers advertise their presence together with various link and Internet parameters either periodically, or in response to a Router Solicitation message.

·· Neighbor Solicitation (Type 135)

·· Neighbor solicitations are used by nodes to determine the link layer address of a neighbor, or to verify that a neighbor is still reachable via a cached link layer address.

·· Neighbor Advertisement (Type 136)

·· Neighbor advertisements are used by nodes to respond to a Neighbor Solicitation message.

·· Redirect (Type 137)

·· Routers may inform hosts of a better first hop router for a destination.

Configure, verify, and troubleshoot STP protocols Describe supported STP modes and interop techniques

Traditional Linux bridges Per VLAN Spanning Tree (PVST) creates a spanning tree instance for a bridge. Rapid PVST (PVRST) supports RSTP enhancements for each spanning tree instance. To use PVRST with a traditional bridge, you must create a bridge corresponding to the untagged native VLAN and all the physical switch ports must be part of the same VLAN. When connected to a switch that has a native VLAN configuration, the native VLAN must be configured to be VLAN 1 only for maximum interoperability.

VLAN-aware bridges only operate in RSTP mode. STP bridge protocol data units (BPDUs) are transmitted on the native VLAN. If a bridge running RSTP (802.1w) receives a common STP (802.1D) BPDU, it falls back to 802.1D operation automatically. RSTP interoperates with MST seamlessly, creating a single instance of spanning tree, which transmits BPDUs on the native VLAN. RSTP treats the MST domain as one giant switch.

When connecting a VLAN-aware bridge to a proprietary PVST+ switch using STP, VLAN 1 must be allowed on all 802.1Q trunks that interconnect them, regardless of the configured native VLAN. This is because only VLAN 1 enables the switches to address the BPDU frames to the IEEE multicast MAC address.

Configuration

Most STP parameters are blacklisted in the ifupdown_blacklist section of the /etc/netd.conf file. Before you configure those parameters, you must edit the file to remove them from the blacklist. A full list of parameters — https://docs.cumulusnetworks.com/display/DOCS/Spanning+Tree+and+Rapid+Spanning+Tre e#SpanningTreeandRapidSpanningTree-paramsSpanningTreeParameterList.

Some of the more commonly edited parameters or functions are STP being turned on, off, have its priority changed, and manipulate ports states and functions via configuration.

Priority

The bridge with the lowest priority is elected the root bridge. The priority must be a number between 0 and 61440 and must be a multiple of 4096; the default is 32768.

BPDU guard

To protect the spanning tree topology from unauthorized switches affecting the forwarding path, configure BPDU guard (Bridge Protocol Data Unit). One common example is when a new switch is connected to an access port off of a leaf switch. If this new switch is configured with a low priority, it could become the new root switch and affect the forwarding path for the entire layer 2 topology. Recovery of the port requires it to be shut down and re-enabled, but the event will reoccur until the cause of the issue is corrected on the connected device.

Below is an example of the error message in /var/log/syslog for a BPDU Guard event.

mstpd: error, MSTP _ IN _ rx _ bpdu: bridge:bond0 Recvd BPDU on BPDU Guard Port - Port Down

Port Admin Edge

Port Admin Edge is equivalent to the PortFast feature offered by other vendors. It enables or disables the initial edge state of a port. Ports configured with PortAdminEdge bypass the listening and learning states to move immediately into forwarding. Using PortAdminEdge mode has the potential to cause loops if not accompanied by the BPDU guard feature. It is common, but not required, for edge ports to be configured as access ports for a simple end host. In the data center, edge ports mostly connect to servers, which might pass both tagged and untagged traffic.

Port Auto Edge

PortAutoEdge is an enhancement to the standard PortAdminEdge (PortFast) mode, which allows for the automatic detection of edge ports. PortAutoEdge enables and disables the auto transition to/from the edge state of a port in a bridge.

When a BPDU is received on port configured with portautoedge, the port ceases to be in edge port state and transitions into a normal STP port. When BPDUs are no longer received on the interface, the port becomes an edge port, and transitions through the discarding and learning states before resuming forwarding.

PortAutoEdge is enabled by default in Cumulus Linux.

Port BPDU filter

You can enable bpdufilter on a switch port, which filters BPDUs in both directions effectively disabling STP. Using BDPU filter inappropriately can cause layer 2 loops. Use this feature deliberately and with extreme caution.

Port network (bridge assurance)

On a point-to-point link where RSTP is running, if you want to detect unidirectional links and put the port in a discarding state (in error), enable bridge assurance by configuring port type network. The port will be in bridge assurance inconsistent state until a BPDU is received from the peer. You need to configure the port

type network on both the ends of the link in order for bridge assurance to operate properly. The default setting for bridge assurance is off.

An example of an error message in /var/log/syslog when a bridge assurance event occurs.

cu mulus@switch:~$ sudo grep -in assurance /var/log/syslog | grep mstp 1365:Jun 25 18:03:17 mstpd: br1007:swp1.1007 Bridge assurance inconsistent

Storm control

Storm control provides protection against excessive inbound BUM (broadcast, unknown unicast, multicast) traffic on layer 2 switch port interfaces, by limiting the packet rate. Configure storm control for each physical port by configuring switchd. To enable unicast and multicast storm control at 400pps and 3000pps, for swp1:

cu mulus@switch:~$ sudo sh -c ‘echo 400 > /cumulus/switchd/config/interface/swp1/storm _ control/unknown _ unicast’ cu mulus@switch:~$ sudo sh -c ‘echo 3000 > /cumulus/switchd/config/interface/swp1/storm _ control/multicast’

Verification

cu mulus@leaf01:m g mt-vrf:~$ net show bridge spanning-tree peerlink bridge:peerlink CIST info enabled yes role Designated port id F.003 state forwarding external port cost 10000 admin external cost 0 internal port cost 10000 admin internal cost 0 designated root 8.000.44:39:39:FF:40:94 dsgn external cost 0 dsgn regional root 8.000.44:39:39:FF:40:94 dsgn internal cost 0 designated bridge 8.000.44:39:39:FF:40:94 designated port F.003 admin edge port no auto edge port yes oper edge port yes topology change ack no point-to-point yes admin point-to-point auto restricted role no restricted TCN no port hello time 2 disputed yes bpdu guard port no bpdu guard error no network port no BA inconsistent no Num TX BPDU 3 Num TX TCN 0 Num RX BPDU 1 Num RX TCN 0 Num Transition FWD 2 Num Transition BLK 1 bpdufilter port no

clag ISL yes clag ISL Oper UP yes clag role primary clag dual conn mac 00:00:00:00:00:00 clag remote portID F.FFF clag system mac 44:39:39:FF:40:94 cu mulus@leaf01:m g mt-vrf:~$ net show bridge spanning-tree vni13 bridge:vni13 CIST info enabled yes role Designated port id 8.004 state forwarding external port cost 2000 admin external cost 0 internal port cost 2000 admin internal cost 0 designated root 8.000.44:39:39:FF:40:94 dsgn external cost 0 dsgn regional root 8.000.44:39:39:FF:40:94 dsgn internal cost 0 designated bridge 8.000.44:39:39:FF:40:94 designated port 8.004 admin edge port yes auto edge port yes oper edge port yes topology change ack no point-to-point yes admin point-to-point auto restricted role no restricted TCN no port hello time 2 disputed no bpdu guard port yes bpdu guard error no network port no BA inconsistent no Num TX BPDU 0 Num TX TCN 0 Num RX BPDU 0 Num RX TCN 0 Num Transition FWD 2 Num Transition BLK 1 bpdufilter port yes clag ISL no clag ISL Oper UP no clag role primary clag dual conn mac 00:00:00:00:00:00 clag remote portID F.FFF clag system mac 44:39:39:FF:40:94

Troubleshooting

The purpose of STP is to detect loops and block forwarding on ports to prevent the loop. STP is broken when it cannot accomplish this task and unmitigated loop occurs. General steps for STP loops.

·· Identify forwarding loop as problem

·· Scope the loop to involved ports

·· Break the loop

·· Fix the cause of the loop

Configure and verify layer 2 protocols Multi-chassis Link Aggregation (MLAG)

https://docs.cumulusnetworks.com/display/DOCS/Multi-Chassis+Link+Aggregation+-+MLAG

cumulus@oob-mgmt-server:~$ net example clag basic-clag Scenario ======------| | swp3 swp3 | | | switch1 |======| switch2 | | mgmt=10.0.0.1 | swp4 swp4 | mgmt=10.0.0.2 | ------| swp1 | swp1 | ------| | | | | ------| host-11 |------| | ------We want to create an MLAG peering relationship between switch1 and switch2 on their swp3 and swp4 interfaces, create vlans 100-200, and dual connect host-11 to swp1 on both switch1 and switch2.

You will need to configure switch1 and switch2, the steps for both are very similar.

- create the peering; select one switch to be primary and the other secondary - backup-ip is an optional (recommened) IP address that is separately reachable - create VLANs 100-200 - configure a host facing interface for clag - switch1 and switch2 MUST use the same clag-id for host-11 - connect the clag to host-11 to vlan 100 untagged - review and commit

net commands ======switch1# net add clag peer sys-mac 44:38:39:FF:01:01 interface swp3-4 primary backup- ip 10.0.0.2 switch1# net add vlan 100-200 switch1# net add clag port bond bond-to-host-11 interface swp1 clag-id 1 switch1# net add bond bond-to-host-11 bridge access 100 switch1# net pending

switch1# net commit

switch2# net add clag peer sys-mac 44:38:39:FF:01:01 interface swp3-4 secondary backup-ip 10.0.0.1 switch2# net add vlan 100-200 switch2# net add clag port bond bond-to-host-11 interface swp1 clag-id 1 switch2# net add bond bond-to-host-11 bridge access 100 switch2# net pending switch2# net commit

Verification ======switch1# net show interface switch1# net show clag

MLAG with STP

Cumulus Networks recommends that you always enable STP in your layer 2 network. With MLAG, Cumulus Networks recommends you enable BPDU guard on the host-facing bond interfaces. Best Practices for STP with MLAG:

·· The STP global configuration must be the same on both the switches

·· The STP configuration for dual-connected ports should be the same on both peer switches

·· The STP priority must be the same on both peer switches MLAG log file

By default, when clagd is running, it logs its status to the /var/log/clagd.log file and syslog.

cu mulus@spine01:~$ sudo tail /var/log/clagd.log 2016-10-03T20:31:50.471400+00:00 spine01 clagd[1235]: Initial config loaded 2016-10-03T20:31:52.479769+00:00 spine01 clagd[1235]: The peer switch is active. 2016-10-03T20:31:52.496490+00:00 spine01 clagd[1235]: Initial data sync to peer done. 2016-10-03T20:31:52.540186+00:00 spine01 clagd[1235]: Role is now primary; elected 2016-10-03T20:31:54.250572+00:00 spine01 clagd[1235]: HealthCheck: role via backup is primary 2016-10-03T20:31:54.252642+00:00 spine01 clagd[1235]: HealthCheck: backup active 2016-10-03T20:31:54.537967+00:00 spine01 clagd[1235]: Initial data sync from peer done. 2016-10-03T20:31:54.538435+00:00 spine01 clagd[1235]: Initial handshake done. 2016-10-03T20:31:58.527464+00:00 spine01 clagd[1235]: leaf03-04 is now dual connected. 2016-10-03T22:47:35.255317+00:00 spine01 clagd[1235]: leaf01-02 is now dual connected.

Describe ethernet bridging fundamentals

https://docs.cumulusnetworks.com/display/DOCS/Ethernet+Bridging+-+VLANs

Ethernet bridges provide a means for hosts to communicate through layer 2, by connecting all of the physical and logical interfaces in the system into a single layer 2 domain. The bridge is a logical interface with a MAC address and an MTU. The bridge MTU is the minimum MTU among all its members. By default, the bridge’s MAC address is copied from eth0. The bridge can also be assigned an IP address.

Bridge members can be individual physical interfaces, bonds or logical interfaces that traverse an 802.1Q VLAN trunk.

Describe & configure connectivity to the host Describe common host attachment modes

Single attached

The server is connected to a single switch for connectivity. The loss of the network devices brings the connected servers down and it is the responsibility of the server and application infrastructure to overcome the loss.

cumulus@switch:~$ net add bridge bridge ports swp1 cumulus@switch:~$ net add bridge bridge vids 10,20 cumulus@switch:~$ net add bridge bridge pvid 1 cumulus@switch:~$ net pending cumulus@switch:~$ net commit

Bonding: active/standby, active/active

Linux bonding provides a method for aggregating multiple network interfaces (slaves) into a single logical bonded interface (bond). Cumulus Linux supports two bonding modes:

·· IEEE 802.3ad link aggregation mode, which allows one or more links to be aggregated together to form a link aggregation group (LAG), so that a media access control (MAC) client can treat the link aggregation group as if it were a single link. IEEE 802.3ad link aggregation is the default mode

·· Balance-xor mode, where the bonding of slave interfaces are static and all slave interfaces are active for load balancing and fault tolerance purposes. This is useful for MLAG deployments

The benefits of link aggregation include:

·· Linear scaling of bandwidth as links are added to LAG

·· Load balancing

·· Failure protection

Cumulus Linux uses version 1 of the LAG control protocol (LACP).

A server can be bonded to a single switch or bonded to multiple switches. Bonding to a single switch provide all links in the bond active, but there is lacking network level redundancy for the connection. A single switch failure will bring the server offline. Traditionally when connecting to multiple switches a server would use a version of NIC teaming to provide redundancy, failover, and outbound from server load sharing. This was dependent on server manufacturer with each having different mechanisms. For a host to achieve active/active forwarding at layer 2 to multiple switches, Multi-chassis Link Aggregation must be used.

Describe the purpose of Multi-chassis Link Aggregation (MLAG)

A MLAGs purpose is to reduce the dependence on spanning tree and enable 100% bandwidth utilization while providing redundancy and failover at both the server and network portions of the access layer connections.

Multi-Chassis Link Aggregation (MLAG), enables a server or switch with a two-port bond, such as a link aggregation group/LAG, EtherChannel, port group or trunk, to connect those ports to different switches and operate as if they are connected to a single, logical switch. This provides greater redundancy and greater system throughput.

Dual-connected devices or hosts can create LACP bonds that contain links to each physical switch. Therefore, active-active links from the dual-connected devices are supported even though they are connected to two different physical switches.

Routing fundamentals

Describe BGP and how it is used Border Gateway Protocol (BGP) overview

BGP is a path-vector routing protocol defined in RFC4271 where each organization control is referenced as a piece of the path. The path consists of the listed numbers of the organizations called autonomous system numbers needed to reach the destination. The shortest path is considered the best route, and BGP uses attributes to exchange paths, origins, next-hops, and other preference settings in order to manipulate and filter route prefixes. BGP utilizes TCP port 179 for building neighbor relationships and exchanging information.

BGP transitions through the following neighbor states:

Neighbor State Summary Reason

Idle Neighbor not responding

Active Attempting to connect to neighbor

Connect TCP connection established

Open Sent Sent open message to neighbor

Open Confirm Response to open message received

Established Neighbor adjacency established

Common BGP attributes:

Name Description

Origin IGP, EGP, or unknown

AS Path List of autonomous systems which the advertisement has traversed

Next Hop Hop for the destination

Local Preference Metric for internal neighbors to reach external destinations

Atomic Aggregate Includes ASes which have been dropped due to route aggregation

Aggregator IS and AS of summarizing router

Community Route tag

Multiple Exit Discriminator (MED) Metric for external neighbros to reach the local AS default 0

Weight Cisco proprietary, not communicated to peers

Default path selection order:

Attribute Preference Description

Weight Highest Cisco only, referenced for interoperability

Local Preference Highest Communicated between peers within an AS

Self Originated Prefer paths originated locally

AS-Path Shortest Minimize AS “hops”

Origin IGP Prefer IGP- learned routes over routes learned over EGP and EGP over unknown

MED Lowest Used for externally entering an AS

External eBGP Prefer external versus internal BGP

IGP Cost Lowest Consider and prefer lower IGP metric

eBGP Peering Oldest Favor more stable routes

Router ID Lowest General tiebeaker if needed

Traditionally BGP is an external gateway protocol intended for use between different autonomous systems, or networks and is routing protocol used between ISPs, but carries a significant use case inside data centers.

Describe the differences between AS placements EBGP vs. IBGP iBGP

iBGP represents an internal BGP connection which is signified by peering 2 devices in the same autonomous system. Since the autonomous system represents a control organization, the systems behave differently for external and internal connections.

Routes learned from iBGP peers will be only be advertised to eBGP peers, and a full mesh between peers is required. Route reflectors and confederations are methods to improve scalability of iBGP and the default full mesh requirement.

eBGP

eBGP represents an external BGP connection signified by the peering devices having different autonomous system numbers. EBGP peers are assumed to be directly connected by default.

Routes learned from an eBGP peer will be advertised to other peers, both iBGP and eBGP.

BGP placement and usage within the data center has evolved over recent years with the scale required. eBGP has shown to be a capable and preferred routing protocol to solve current challenges to data center scale and automation improvements. Dinesh Dutt’s book provides a good look into these scenarios and choices. https://cumulusnetworks.com/lp/bgp-ebook/

Describe how OSPF is used and LSA types Open Shortest Path First (OSPF) overview

https://docs.cumulusnetworks.com/display/DOCS/Open+Shortest+Path+First+-+OSPF

https://en.wikipedia.org/wiki/Link-state_advertisement

OSPF is an interior gateway protocol using a link-state routing algorithm (Dijkstra) defined in RFC2328 with updates for IPv6 with OSPFv3 in RFC5340. OSPF uses multicast groups for neighbor discovery and hellos and supports MD5 and AH (v3) authentication.

OSPF maintains the view of the network topology conceptually as a directed graph. Each router represents a vertex in the graph. Each link between neighboring routers represents a unidirectional edge and each link has an associated weight (called cost) that is either automatically derived from its bandwidth or administratively assigned. Using the weighted topology graph, each router computes a shortest path tree (SPT) with itself as the root, and applies the results to build its forwarding table. The computation is generally referred to as SPF computation and the resultant tree as the SPF tree.

An LSA (link-state advertisement) is the fundamental quantum of information that OSPF routers exchange with each other. It seeds the graph building process on the node and triggers SPF computation. LSAs originated by a node are distributed to all the other nodes in the network through a mechanism called flooding. Flooding is done hop-by-hop. OSPF ensures reliability by using link state acknowledgement packets. The set of LSAs in a router’s memory is termed link-state database (LSDB), a representation of the network graph. Therefore, OSPF ensures a consistent view of LSDB on each node in the network in a distributed fashion (eventual consistency model); this is key to the protocol’s correctness.

Type Name Description

Type 1 Router LSA The Router LSA is generated by each router for each area it is located. In the link- state ID you will find the originating router’s ID.

Type 2 Network LSA Network LSAs are generated by the DR. The link-state ID will be the router ID of the DR.

Type 3 Summary LSA The summary LSA is created by the ABR and flooded into other areas.

Type 4 Summary Other routers need to know where to find the ASBR. This is why the ABR will ASBR LSA generate a summary ASBR LSA which will include the router ID of the ASBR in the link-state ID field.

Type 5 External LSA The external LSAs are generated by the ASBR.

Type 6 Multicast LSA Not supported and not used.

Type 7 External NSSA An external LSA for NSSA (not-so-stubby-area) which doesn’t allow external LSAs (type 5).

LSA

OSPF as a DC underlay

EVPN can be deployed with an OSPF or static route underlay if needed. This is a more complex configuration than using eBGP. In this case, OSPF is responsible for neighboring and exchanging loopback routing information. The loopbacks are used to peer for the overlay. A separate routing instance or protocol is then required for the overlay. OSPF is IPv4 only and OSPFv3 may not support IPv4 in all implementations.

Cumulus Linux supports IP unnumbered interfaces for OSPF over Ethernet point-to-point interfaces and recommends using Prescriptive Topology Manager to verify link connectivity in this scenario. IP unnumbered can conserve address space in the network, reduce LSA table size (using less memory) as well as make automation easier for larger networks.

OSPF area placement

Each pod of a two-tier Clos network are assigned an area. For a single pod, all devices are in area 0. LSA Types 1 and 2 are never flooded outside of their local areas.

The area border router (ABR) is placed on the spine switches. The connectivity to and from the data center can be either in the pod’s area or moved to area 0. For ease of automation, the non-backbone areas could be configured with the same area number in this scenario.

Beyond two-tiers, the massively scalable data center, each super-spine switch would be in area 0. Area 0 can be discontiguous if the switches never need to talk with each other. Each pod would be in its own area so the SPF calculations would be limited to its local pod.

OSPF stub areas

https://docs.cumulusnetworks.com/display/DOCS/Open+Shortest+Path+First+- +OSPF#OpenShortestPathFirst-OSPF-StubAreas

Stub areas help improve scalability for larger networks reducing LSAs types. Type 5 External LSAs can take up a large percentage of the database size.

Area Type LSA Behavior

Normal non-zero area LSA types 1, 2, 3, 4 area-scoped, type 5 externals, inter-area routes summarized

Stub area LSA types 1, 2, 3, 4 area-scoped, No type 5 externals, inter-area routes summarized

Totally stubby area LSA types 1, 2 area-scoped, default summary, No type 3, 4, 5 LSA types allowed

Describe the components of FHRP Switched virtual interface

Bridges can be included as part of a routing topology once assigned an IP address. This enables hosts within the bridge to communicate with other hosts outside of the bridge, via a switch virtual interface (SVI), which provides layer 3 routing. The IP address of the bridge is typically from the same subnet as the bridge’s member hosts.

FIGURE 6

Virtual Router Redundancy (VRR)

VRR enables hosts to communicate with any redundant router without reconfiguration, running dynamic router protocols, or running router redundancy protocols. Redundant routers will respond to ARP requests from hosts in an identical manner, but if one fails, the other redundant routers will continue to respond, leaving the hosts with the impression that nothing has changed. Cumulus Linux only supports VRR on switched virtual interfaces (SVIs). VRR is NOT supported on physical interfaces or virtual subinterfaces.

As the bridges in each of the redundant routers are connected, they will each receive and reply to ARP requests for the virtual router IP address.

A range of MAC addresses is reserved for VRR to prevent MAC address conflicts with other interfaces in the same bridged network. The reserved range is 00:00:5E:00:01:00 to 00:00:5E:00:01:ff. Cumulus Networks recommends using MAC addresses from the reserved range when configuring VRR. The reserved MAC address range for VRR is the same as for the Virtual Router Redundancy Protocol (VRRP), as they serve similar purposes. VRRP is separate but can be configured instead, if preferred.

cumulus@switch:~$ net add bridge cumulus@switch:~$ net add vlan 500 ip address 192.0.2.252/24 cumulus@switch:~$ net add vlan 500 ip address-virtual 00:00:5e:00:01:01 192.0.2.254/24 cumulus@switch:~$ net add vlan 500 ipv6 address 2001:db8::1/32 cumulus@switch:~$ net add vlan 500 ipv6 address-virtual 00:00:5e:00:01:01 2001:d b8::f/32 cumulus@switch:~$ net pending cumulus@switch:~$ net commit

The VLAN interface must have unique IP addresses for both the physical (the address option) and virtual (the address-virtual option) interfaces, as the unique address is used when the switch initiates an ARP request.

https://docs.cumulusnetworks.com/display/DOCS/Virtual+Router+Redundancy+-+VRR+and+VRRP#Virtual RouterRedundancy-VRRandVRRP-VRR

Virtual Router Redundancy Protocol (VRRP)

https://docs.cumulusnetworks.com/display/DOCS/Virtual+Router+Redundancy+-+VRR+and+VRRP#Virtual RouterRedundancy-VRRandVRRP-VRRP

Virtual Router Redundancy Protocol (VRRP) allows for a single virtual default gateway shared among two or more network devices in active/standby. The VRRP master forwards packets, and if the master VRRP router fails, another VRRP standby router automatically takes over the master role. VRRP advertisements are sent to other VRRP routers in the same virtual router group, which include priority and state. VRRP router priority determines the role that each virtual router plays and who becomes the new master if the master fails.

All virtual routers use 00:00:5E:00:01:XX for IPv4 gateways and 00:00:5E:00:02:XX for IPv6 gateways as their MAC address. The last byte of the address is the Virtual Router IDentifier (VRID), which is different for each virtual router group in the network. This MAC address is used by only one physical router at a time, which replies with this address when ARP requests or neighbor solicitation packets are sent for the IP addresses of the virtual router.

·· VRRP is supported starting in Cumulus Linux 3.7.4

·· Cumulus Linux supports both VRRPv2 and VRRPv3. The default protocol version is VRRPv3

·· 255 virtual routers are supported per switch

·· VRRP is not supported currently in an MLAG environment or with EVPN

·· VRRP is supported on physical interfaces and sub-interfaces

FIGURE 7

cumulus@spine01:~$ net add interface swp1 vrrp 44 10.0.0.1/24 cumulus@spine01:~$ net add interface swp1 vrrp 44 2001:0db8::1/64 cumulus@spine01:~$ net add interface swp1 vrrp 44 priority 254 cumulus@spine01:~$ net add interface swp1 vrrp 44 advertisement-interval 5000 cumulus@spine01:~$ net pending cumulus@spine01:~$ net commit

cumulus@spine02:~$ net add interface swp1 vrrp 44 10.0.0.1/24 cumulus@spine02:~$ net add interface swp1 vrrp 44 2001:0db8::1/64 cumulus@spine02:~$ net pending cumulus@spine02:~$ net commit cu mulus@spine01:~$ net show vrrp 44 Virtual Router ID 44 Protocol Version 3 Autoconfigured No Shutdown No Interface swp1 VRRP interface (v4) vrrp4-3-1 VRRP interface (v6) vrrp6-3-1 Primary IP (v4) Primary IP (v6) fe80::54df:e543:5c12:7762 Virtual MAC (v4) 00:00:5e:00:01:01 Virtual MAC (v6) 00:00:5e:00:02:01

Status (v4) Master Status (v6) Master Priority 254 Effective Priority (v4) 254 Effective Priority (v6) 254 Preempt Mode Yes Accept Mode Yes Advertisement Interval 5000 ms Master Advertisement Interval (v4) 0 ms Master Advertisement Interval (v6) 5000 ms Advertisements Tx (v4) 17 Advertisements Tx (v6) 17 Advertisements Rx (v4) 0 Advertisements Rx (v6) 0 Gratuitous ARP Tx (v4) 1 Neigh. Adverts Tx (v6) 1 State transitions (v4) 2 State transitions (v6) 2 Skew Time (v4) 0 ms Skew Time (v6) 0 ms Master Down Interval (v4) 0 ms Master Down Interval (v6) 0 ms IPv4 Addresses 1 ...... 10.0.0.1 IPv6 Addresses 1 ...... 2001:0db8::1

Anycast gateway

An anycast gateway is used with VXLAN routing typically in a distributed routing architecture. A distributed architecture involves configuring an SVI and enabling VXLAN routing on each leaf switch. Therefore, VXLAN routing occurs closest to the host, keeping traffic local for more efficient routing and lower latency than the centralized architecture.

Using an Anycast gateway, every leaf’s SVI is configured with the same IP address per VLAN. Since all hosts within a VLAN are configured with the same IP default gateway address, all hosts or VMs can be easily moved throughout the data center without changing their configuration.

Describe the components of a routing table Describe Equal Cost Multipath (ECMP) routing

ECMP routing is when multiple next hops are installed to the same destination, due to the same protocol containing multiple routes with an identical cost or metric. ECMP is enabled by default in Cumulus Linux and load sharing occurs automatically for all routes with multiple next hops installed. ECMP load sharing supports both IPv4 and IPv6 routes.

Describe hashing

To prevent out of order packets, ECMP hashing is done on a per-flow basis, which means that all packets with the same source and destination IP addresses and the same source and destination ports always hashed to the same next hop. ECMP hashing does not keep a record of flow states, nor a record of packets, nor guarantees that traffic sent to each next hop is equal.

Define equal cost

For routes to be considered equal cost they must:

·· Be the identical route, including network and prefix length. A /24 and /25 are NOT the same route.

·· Originate from the same routing protocol. Routes from different sources are not considered equal. For example, a static route and an OSPF route are not considered for ECMP load sharing.

·· Have equal cost or metric within the routing protocol. If two routes from the same protocol are unequal, only the best route is installed in the routing table.

Comparing different sources like OSPF and BGP routes

When different protocols provide the same route prefix, their administrative distance is compared to determine which route should be utilized with the lower distance preferred.

Default distance examples:

·· eBGP: 20

·· iBGP: 200

·· OSPF: 110

·· RIP: 120

In the example below, BGP and OSPF are providing routes for 10.0.0.11/32, 10.0.0.21/32, and 10.0.0.22/32 which are the loopback interfaces of leaf01, spine01 and spine02. Longest prefix length match is chosen prior to comparison protocols sources. A /32 matching route will be preferred over /30 regardless of protocol source for example.

cumulus@leaf02:mgmt-vrf:~$ net show route ipv4 Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, > - selected route, * - FIB route

O 10.0.0.11/32 [110/200] via 10.0.0.21, swp51 onlink, 00:00:53 via 10.0.0.22, swp52 onlink, 00:00:53 B>* 10.0.0.11/32 [20/0] via fe80::4638:39ff:fe00:602, swp51, 00:00:57 * via fe80::4638:39ff:fe00:702, swp52, 00:00:57 O 10.0.0.12/32 [110/0] is directly connected, lo, 00:05:34

C * 10.0.0.12/32 is directly connected, swp52, 00:05:35 C * 10.0.0.12/32 is directly connected, swp51, 00:05:35 C>* 10.0.0.12/32 is directly connected, lo, 00:05:35 B>* 10.0.0.13/32 [20/0] via fe80::4638:39ff:fe00:602, swp51, 00:00:57 * via fe80::4638:39ff:fe00:702, swp52, 00:00:57 B>* 10.0.0.14/32 [20/0] via fe80::4638:39ff:fe00:602, swp51, 00:00:56 * via fe80::4638:39ff:fe00:702, swp52, 00:00:56 O 10.0.0.21/32 [110/100] via 10.0.0.21, swp51 onlink, 00:03:58 B>* 10.0.0.21/32 [20/0] via fe80::4638:39ff:fe00:602, swp51, 00:05:31 O 10.0.0.22/32 [110/100] via 10.0.0.22, swp52 onlink, 00:00:53 B>* 10.0.0.22/32 [20/0] via fe80::4638:39ff:fe00:702, swp52, 00:00:57 C>* 10.0.0.112/32 is directly connected, lo, 00:05:35 B>* 10.0.0.134/32 [20/0] via fe80::4638:39ff:fe00:602, swp51, 00:00:57 * via fe80::4638:39ff:fe00:702, swp52, 00:00:57 C * 10.1.3.0/24 is directly connected, vlan13-v0, 00:05:35 C>* 10.1.3.0/24 is directly connected, vlan13, 00:05:35 C * 10.2.4.0/24 is directly connected, vlan24-v0, 00:05:35 C>* 10.2.4.0/24 is directly connected, vlan24, 00:05:35 C>* 169.254.1.0/30 is directly connected, peerlink.4094, 00:05:35

Describe how a routing table is populated by different routing information sources Devices use the administrative distance, routing protocol metric, and the prefix length to determine which routes will be placed into the routing table. The routing table is built using the following general steps:

1. If the route entry does not currently exist in the routing table, add it to the routing table

2. If the route entry is more specific than an existing route, add it to the routing table. Both the new and less specific entry are retained in the routing table.

3. If the route entry is the same as an existing one, but is received from a more preferred route source, replace the forwarding entry with the new entry

4. If the route entry is the same as an existing one, and is received from the same protocol:

a. Discard the new route if the metric is higher than the existing route

b. Replace the existing route if the metric of the new route is lower

c. If the metric for both routes is the same, use both routes for load-balancing

The routing sources can populate the routing table:

·· Automatically by the router for connected and local routes

·· Statically by an administrator. The administrator statically configures how to get to remote networks.

·· Dynamically by a routing protocol. The administrator configures the networks this router can get to. The dynamic routing protocol does the work to figure out how to get to remote networks and updates

the database according to the way the dynamic routing protocol works.

FIGURE 8

Route entry displayed components:

·· Route source: Identifies how the route was learned.

·· C — connected

·· S — static

·· R — RIP

·· O — OSPF

·· I — IS-IS

·· B — BGP

·· E — EIGRP

·· Destination network: Identifies the address of the remote network.

·· Administrative distance: Identifies the trustworthiness of the route source.

·· Metric: Identifies the value assigned to reach the remote network. Lower values indicate preferred routes.

·· Next hop: Identifies the IPv4 address of the next router to forward the packet to.

·· Route timestamp: Identifies from when the route was last heard.

·· Outgoing interface: Identifies the exit interface to use to forward a packet toward the final destination.

A device will check for the longest match to select the most specific route to the destination. For example a packet destined for 172.16.0.10 with multiple matching routes will compare prefix lengths. In the example

pictured on the right, the device has three possible routes that match this packet: 172.16.0.0/12, 172.16.0.0/18, and 172.16.0.0/26. Of the three routes, 172.16.0.0/26 has the longest match and is therefore chosen to forward the packet.

FIGURE 9

Compare and contrast static routing and dynamic routing Static routing means the administrator must configure each device with destination forwarding information, taking into account redundancy, failover. Static routes quickly become an administrative issue, even with automation.

Routing protocols dynamically compute reachability to destinations based on information state. Dynamic routing protocols can adjust to changes in the network without administrative interaction. They provide significant advantages for overall uptime and scale to large numbers of devices.

Compare and contrast different routing protocols BGP

An overview for BGP was provided via the earlier section, BGP Overview, and is discussed in detail throughout the study guide.

OSPF

An overview for BGP was provided via the earlier section, OSPF Overview.

Routing Information Protocol (RIP)

RIP is a distance vector protocol defined in RFC1058 (v1), RFC2080 (ipv6 NG), and RFC2453 (v2), which is generally limited to use in smaller networks by its lack of features and scalability, and is declining in overall usage since the 1990s. It is simple and easy to use, but limited to 15 hops. For IPv4, RIPv1 and RIPv2 are

available versions. Both use hop count as a metric, send routing tables every 30 seconds, and are assigned a distance of 120, but version 1 uses broadcasts for updates and cannot advertise subnet masks while version 2 moves to multicast (224.0.0.9) updates and subnet mask support. RIP employs split horizon to help prevent loops by blocking the advertisement of a network on the same interface it was learned on.

·· Update Timer (30 seconds)

·· Defines how often the router will send out a routing table update.

·· Invalid Timer (180 seconds)

·· Indicates how long a route will remain in a routing table before being marked as invalid, if no new updates are heard about this route.

·· The invalid timer will be reset if an update is received for that particular route before the timer expires.

·· A route marked as invalid is not immediately removed from the routing table. Instead, the route is marked with a metric of 16, which means the route is unreachable, and will be placed in a hold-down state.

·· Hold-down Timer (180 seconds)

·· Specifies how long RIP will keep a route from receiving updates when it is in a hold-down state.

·· In a hold-down state, RIP will not receive any new updates for routes until the hold-down timer expires. A route will go into a hold-down state for the following reasons:

·· The invalid timer has expired.

·· An update has been received from another router; route goes into a 16 metric (or unreachable).

·· An update has been received from another router; route goes into a higher metric than what it is currently using.

·· Flush Timer (240 seconds)

·· When no new updates are received about this route, flush timer indicates how long a route can remain in a routing table before getting flushed out.

·· The flush timers operates simultaneously with the invalid timer, so every 60 seconds, after it has been marked invalid, the route will get flushed out.

·· When RIP timer is not in sync with all routers on the RIP network, system instability occurs.

·· This timer must be set to a higher value than the invalid timer.

For modern usage, RIP is generally limited to consumer grade devices and networks in need of an upgrade.

Intermediate System to Intermediate System (IS-IS)

IS-IS is an link-state interior gateway protocol designed for use within an administrative domain defined is ISO/IEC 10589:2002 within the OSI reference design, and sees its primary usage in large service provider networks. IS-IS operates by flooding link state information through a network of devices with each device independently building a database of the network’s topology. Conceptually similar to OSPF, IS-IS LAO uses Dijkstra’s algorithm for best path computation.

While OSPF is a layer 3 protocol and was built to run on IP, IS-IS is a layer 2 protocol with IP support defined in RFC1195.

IS-IS differs from OSPF in the way that “areas” are defined and routed between. IS-IS routers are designated as being: Level 1 (intra-area); Level 2 (inter area); or Level 1–2 (both). Routing information is exchanged between Level 1 routers and other Level 1 routers of the same area, and Level 2 routers can only form relationships and exchange information with other Level 2 routers. Level 1–2 routers exchange information

with both levels and are used to connect the inter area routers with the intra area routers.

In IS-IS area borders are in between routers, designated as Level 2 or Level 1–2. The result is that an IS-IS router is only ever a part of a single area. IS-IS also does not require Area 0 (Area Zero) to be the backbone area through which all inter-area traffic must pass. The logical view is that OSPF creates something of a spider web or star topology of many areas all attached directly to Area Zero and IS-IS by contrast creates a logical topology of a backbone of Level 2 routers with branches of Level 1–2 and Level 1 routers forming the

OSPF IS-IS

Host End System (ES)

Router Intermediate System (IS)

Link Circuit

Packet Protocol Data Unit (PDU)

Designated Router (DR) Designated IS (DIS)

Backup DR (BDR) N/A (no BDIS is Used)

Link-state Advertisement (LSA) Link-state PDU (LSP)

Hello Packet IIH PDU

Database Description (DBD) Complete Sequence Number PDU (CSNP)

Area Sub-domain

Non-backbone Area Level-1 (Station)

Backbone Area Level-2 (Area)

Area Border Router (ABR) L1L2 (Station and Area)

Autonomous System Border Router (ASBR) Any IS

individual areas.

As of Cumulus Linux 3.7.3, NCLU does not support interacting with IS-IS and it must be configured through direct FRR manipulation. http://docs.frrouting.org/en/latest/isisd.html

Terminology Comparison between OSPF and IS-IS:

FIGURE 10

Enhanced Interior Gateway Routing Protocol (EIGRP)

EIGRP is a hybrid distance vector routing protocol developed by Cisco Systems, which remained unique to Cisco equipment until 2013 when it was published in an IETF draft, then later published as RFC7868 in 2016, and adopted by some vendors.

EIGRP utilizes multicast (224.0.0.10) for hello packets to form and maintain neighbor adjacencies. It utilizes bandwidth and delay as metrics by default, with the option to enable load, reliability, and MTU, as well as

customize their weights in metric calculation through K value manipulation. In order for neighbor to form an adjacency they must agree on ASN, subnet, and K values. The composite metric formula for EIGRP is displayed below.

FIGURE 11

Quick comparison chart

FIGURE 12

*Chart shows older Cisco proprietary information for EIGRP, rather than RFC7868 *Is IS-IS distance right for Cumulus Linux?

Describe IPv4 and IPv6 addressing fundamentals IPv4 addressing overview

Internet Protocol version 4 is a core protocol for standards based networking on the internet and still routes most traffic today despite the ongoing deployment of IPv6, and exhaustion of new IPv4 assignments. IPv4 is described in RFC791 and uses 32 bit addressing providing a maximum of 4,294,967,296 (~4.295 billion) IP addresses, although ~590 million are reserved for various purposes. The addressing is usually represented in dot-decimal format (10.10.10.1) with 4 decimals separated by a period.

FIGURE 13

Binary Dot-decimal notation

Network 11000000.10101000.00000100.00000000 192.168.4.0

Broadcast 11000000.10101000.00000101.11111111 192.168.5.255

IPv4 broadcast

The IP address is combined with a subnet mask to determine the size and scope for the network or subnet, as well as determine network and broadcast addresses and the available hosts included. The network address is the constant network space with the host portion with all zeros. The broadcast address is the constant network space with the host portion space with all ones. The network and host portion do not have to reside on the dotted bit boundary, as shown in the example for 192.168.5.0/23 network is below. Broadcast addresses distribute the layer 3 packet to all hosts in the given network. Broadcasts are traditionally filtered by routers, and need forwarding to reach an off network address. A good example of this is DHCP and DHCP relay. Router take the broadcast and forward it via unicast towards the specific DHCP server.

The broadcast address can be the last address in a network for directing a broadcast to a specific network, but more commonly is all 1s (255.255.255.255) when used.

IPv4 unicast

An ipv4 unicast address is used to send information to a specific host. Any IP address in the 192.168.4.0/23 network above that is not the network or broadcast address is a unicast address. 192.168.4.1 through 192.168.5.254. For multiple receivers to get the same data, the data must be sent once for each and e very receiver.

IPv4 multicast

Multicast addresses are design to deliver data to multiple receivers at once who are subscribed and joined to the multicast information stream. A good example of this is IPTV video services offered by ISPs, where set top boxes subscribe to the multicast stream of a specific IPTV channel. They are also commonly used in routing protocol neighbor discovery and peering relationships such as OSPF, IS-IS, and EIGRP.

Multicast addresses reside in the 224.0.0.0/4 (224.0.0.0 through 239.255.255.255) range and is carved up within that range for specific reserved purposes.

IPv6 overview

Internet Protocol version 6 is an updated version in IP currently undergoing deployment. IPv6 addressing is described in RFC4291, and offers 128 bit addressing providing a maximum of ~3.4*10^38 addresses. The address is usually represented in 32 hexadecimal characters, separated by colons “:” into 8 groups of 4 with extraneous zeros removed. 2001:0db8:85a3:0000:0000:8a2e:0370:7334 becomes 2001:db8:85a3::8a2e:370:7334.

FIGURE 14

Unicast and anycast addresses are typically composed of two logical parts: a 64-bit network prefix used for routing, and a 64-bit interface identifier used to identify a host’s network interface. The network prefix (the routing prefix combined with the subnet id) is contained in the most significant 64 bits of the address. The size of the routing prefix may vary; a larger prefix size means a smaller subnet id size. The bits of the subnet id field are available to the network administrator to define subnets within the given network. The 64-bit interface identifier is either automatically generated from the interface’s MAC address using the modified EUI-64 format, obtained from a DHCPv6 server, automatically established randomly, or assigned manually.

IPv6 unicast

Bits 48 (Or More) 16 (Or Fewer) 64

Field Routing Prefix Subnet ID Interface Identifier

IPv6 link local

Bits 10 54 64

Field Prefix Zeroes Interface Identifier

The prefix field contains the binary value 1111111010. The 54 zeroes that follow make the total network prefix the same for all link-local addresses (fe80::/64 link-local address prefix), rendering them non-routable.

IPv6 multicast

Multicast addresses are formed according to several specific formatting rules, depending on the application, but the general format is below. IPv6 does not use broadcast addresses as IPv4 does, rather using the specially defined all-nodes multicast address.

Bits 8 4 4 112

Field Prefix FLA SC Group ID

Configure, verify, and troubleshoot IPv4 and IPv6 static routing IPv4 static route configuration

https://docs.cumulusnetworks.com/display/DOCS/Routing

Static routes are managed using NCLU or the Cumulus Linux ip route command. The routes are added to the FRRouting routing table, and are then updated into the kernel routing table as well.

cumulus@switch:~$ net add routing route 203.0.113.0/24 198.51.100.2

IPv4 static route verification

Static routes can be verified via NCLU show commands focused to their scope.

cu mulus@switch:~$ net show route static RIB entry for static ======Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, > - selected route, * - FIB route S>* 203.0.113.0/24 [1/0] via 198.51.100.2, swp3

IPv6 static route configuration

IPv6 static routes are configured in the same manner as IPv4 via NCLU.

cumulus@oob-mgmt-server:~$ net add routing route ffff:ffff:ffff::0/48 lo

IPv6 static route verification

cu mulus@switch:~$ net show route static RIB entry for static ======Codes: K - kernel route, C - connected, S - static, R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, > - selected route, * - FIB route S>* ffff:ffff:ffff::/48 [1/0] is directly connected, lo, 00:05:36

Describe the Linux theory on VRF Virtual Routing and Forwarding (VRF) overview

https://docs.cumulusnetworks.com/display/DOCS/Virtual+Routing+and+Forwarding+-+VRF

Cumulus contributed VRFs to the Linux code. VRFs are individual logical routers running on a device to logically segment layers 3 routing tables. This is useful for multitenancy, leveraging powerful resources efficiently, and separating management and data plane routing tables. VRF is fully supported in the Linux kernel, and has the following characteristics:

·· The VRF is presented as a layer 3 master network device with its own associated routing table.

·· The layer 3 interfaces (VLAN interfaces, bonds, SVIs) associated with the VRF are enslaved to that VRF; IP rules direct FIB (forwarding information base) lookups to the routing table for the VRF device.

·· The VRF device can have its own IP address, known as a VRF-local loopback.

·· Applications can use existing interfaces to operate in a VRF context — by binding sockets to the VRF device or passing the ifindex using cmsg. By default, applications on the switch run against the default VRF. Services started by systemd run in the default VRF unless the VRF instance is used. If management VRF is enabled, logins to the switch default to the management VRF. This provides convenience as users to not have to specify management VRF in each command.

·· Listen sockets used by services are VRF-global by default unless the application is configured to use a more limited scope, such as in the management VRF. Connected sockets (like TCP) are then bound to the VRF domain in which the connection originates. The kernel provides a sysctl that allows a single instance to accept connections over all VRFs. For TCP, connected sockets are bound to the VRF the first packet was received. This sysctl is enabled for Cumulus Linux.

·· Connected and local routes are placed in appropriate VRF tables.

·· Neighbor entries continue to be per-interface, and you can view all entries associated with the VRF device.

·· A VRF does not map to its own network namespace; however, you can nest VRFs in a network namespace.

·· You can use existing Linux tools to interact with it, such as tcpdump.

Cumulus Linux supports up to 255 VRFs on a switch.

Describe MGMT VRF theory

https://docs.cumulusnetworks.com/display/DOCS/Management+VRF

Management VRF is a subset of VRF and provides separation between the out-of-band management network and the in-band data plane network. The main routing table is the default table for all of the data plane switch ports, and a management VRF, mgmt, is used for routing through the Ethernet ports of the switch. The mgmt name is special cased to identify the management VRF from a data plane VRF.

Cumulus Linux only supports eth0 or eth1 as the management interface, depending on the switch platform. The Ethernet ports are software-only and not hardware accelerated by switchd. VLAN subinterfaces, bonds, bridges, and the front panel switch ports are not supported as management interfaces.

When management VRF is enabled, logins to the switch are set into the management VRF context and IPv4 and IPv6 networking applications (for example, Ansible, Chef, and apt-get) run by an administrator

communicate out the management network by default. This default context does not impact services run through systemd and the systemctl command, and does not impact commands examining the state of the switch, such as the ip command to list links, neighbors, or routes.

Configure VRF

You configure VRF by associating each subset of interfaces to a VRF routing table, and configuring an instance of the routing protocol — BGP or OSPFv2 — for each routing table. Configure the VRF using NCLU, then place the layer 3 interface(s) in the VRF.

cumulus@switch:~$ net add vrf rocket vrf-table auto cumulus@switch:~$ net add interface swp1 vrf rocket

Configure management VRF

When you commit the change to add the management VRF, all connections over eth0 are dropped. This can impact any automation that might be running.

cumulus@switch:~$ net add vrf mgmt

VRF verification

cumulus@leaf02:mgmt-vrf:~$ net show vrf

VRF Table ------mgmt 1001 rocket 1002

cumulus@leaf02:mgmt-vrf:~$ net show vrf list

VRF: mgmt ------eth0 UP 44:38:39:00:03:00

VRF: rocket ------

VRF route table

You can view the route table of each VRF independently.

cu mulus@switch:~$ net show route vrf rocket RIB entry for rocket ======Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, T - Table, > - selected route, * - FIB route

C>* 169.254.2.8/30 is directly connected, swp1.2 C>* 169.254.2.12/30 is directly connected, swp2.2 C>* 169.254.2.16/30 is directly connected, swp3.2 cumulus@leaf02:mgmt-vrf:~$ net show route vrf mgmt show ip route vrf mgmt ======Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, > - selected route, * - FIB route

VRF mgmt: K>* 0.0.0.0/0 [0/0] via 192.168.0.254, eth0, 02:16:47 K * 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 02:17:23 C>* 192.168.0.0/16 is directly connected, eth0, 02:16:48

show ipv6 route vrf mgmt ======Codes: K - kernel route, C - connected, S - static, R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, > - selected route, * - FIB route

VRF mgmt: K * ::/0 [255/8192] unreachable (ICMP unreachable), 02:17:23 C>* fe80::/64 is directly connected, eth0, 02:17:23 K>* ff00::/8 [0/256] is directly connected, eth0, 02:17:23

Mcast (no PIM) Describe IGMP functionalities

IGMP (Internet Group Management Protocol) and MLD (Multicast Listener Discovery) snooping are implemented in the bridge driver in the Cumulus Linux kernel and are enabled by default. IGMP snooping processes IGMP v1/v2/v3 reports received on a bridge port in a bridge to identify the hosts which would like to receive multicast traffic destined to that group.

FIGURE 15

When an IGMPv2 leave message is received, a group specific query is sent to identify if there are any other hosts interested in that group, before the group is deleted.

An IGMP query message received on a port is used to identify the port that is connected to a router and is interested in receiving multicast traffic.

MLD snooping processes MLD v1/v2 reports, queries and v1 done messages for IPv6 groups. If IGMP or MLD snooping is disabled, multicast traffic gets flooded to all the bridge ports in the bridge. Similarly, in the absence of receivers in a VLAN, multicast traffic would be flooded to all ports in the VLAN. The multicast group IP address is mapped to a multicast MAC address and a forwarding entry is created with a list of ports interested in receiving multicast traffic destined to that group.

Linux concepts

Describe the basics of GRUB GRUB is a boot loader that understands the underlying file system by maintaining a driver for each file system the operating system supports. This approach eliminates the need for hardcoded locations of hard disk sectors and existence of map files, and does not require Master Boot Record (MBR) updates after the

kernel images are added or moved around. Configuration of a boot loader is stored in a regular file, which is also accessed in a file system-aware way to obtain boot configurations before the actual booting of any kernel images.

Display how to boot a switch, recover a password, and manually boot Restart ONIE installer

Reprovisioning the system deletes all system data from the switch. A reboot is required for the reinstall to begin. To initiate the provisioning and installation process, run the onie-select -i command:

cu mulus@switch:~$ sudo onie-select -i WARNING: WARNING: Operating System install requested. WARNING: This will wipe out all system data. WARNING: Are you sure (y/N)? y Enabling install at next reboot...done. Reboot required to take effect.

To cancel a pending reinstall operation, run the onie-select -c command:

cu mulus@switch:~$ sudo onie-select -c Cancelling pending install at next reboot...done.

Uninstall all images & remove configuration

To remove all installed images and configurations and return the switch to its factory defaults, run the onie-select -k command:

cu mulus@switch:~$ sudo onie-select -k WARNING: WARNING: Operating System uninstall requested. WARNING: This will wipe out all system data. WARNING: Are you sure (y/N)? y Enabling uninstall at next reboot...done. Reboot required to take effect.

Boot into rescue mode

If your system becomes broken is some way, you can correct certain issues by booting into ONIE rescue mode. In rescue mode, the file systems are unmounted and you can use various Cumulus Linux utilities to try and resolve a problem. To reboot the system into ONIE rescue mode, run the onie-select -r command:

cu mulus@switch:~$ sudo onie-select -r WARNING: WARNING: Rescue boot requested. WARNING: Are you sure (y/N)? y Enabling rescue at next reboot...done. Reboot required to take effect.

Password recovery

Use single user mode to assist in troubleshooting system boot issues or for password recovery. To enter single user mode, follow the steps below.

1. Boot the switch, as soon as you see the GRUB menu,

2. Use the ^ and v arrow keys to select Advanced options for Cumulus Linux GNU/Linux. A menu similar to the following should appear:

GNU GRUB version 2.02~beta2-22+deb8u1 +------+ | Cumulus Linux GNU/Linux, with Linux 4.1.0-cl-1-amd64 | | Cumulus Linux GNU/Linux, with Linux 4.1.0-cl-1-amd64 (sysvinit) | |*Cumulus Linux GNU/Linux, with Linux 4.1.0-cl-1-amd64 (recovery mode) | | | +------+

3. Select Cumulus Linux GNU/Linux, with Linux 4.1.0-cl-1-amd64 (recovery mode).

4. Press ctrl-x to reboot.

5. After the system reboots, set a new root password.

cu mulus@switch:~$ sudo passwd Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully

6. Sync the /etc directory using btrfs, then reboot the system:

cu mulus@switch:~$ sudo btrfs filesystem sync /etc cu mulus@switch:~$ sudo reboot -f Restarting the system.

Installing software and package management (.deb, source...etc.) — high-level concepts Understand how to use a change log

A changelog is a record of all notable changes made to a project or software program. Cumulus Linux release notes keep a record of all changes from version to version.

https://support.cumulusnetworks.com/hc/en-us/articles/360007793174-Cumulus-Linux-3-7-Release-Notes

A commit example from the 4.19.2 Linux kernel.

https://cdn.kernel.org/pub/Linux/kernel/v4.x/ChangeLog-4.19.2

commit ab5d01b6130a4faa37a393cf828c6f65c45e7251 Author: David Ahern Date: Wed Oct 24 08:32:49 2018 -0700

net: sched: Remove TCA _ OPTIONS from policy

commit e72bde6b66299602087c8c2350d36a525e75d06e upstream.

Marco reported an error with hfsc: root@Calimero:~# tc qdisc add dev eth0 root handle 1:0 hfsc default 1 Error: Attribute failed policy validation.

Apparently a few implementations pass TCA _ OPTIONS as a binary instead of nested attribute, so drop TCA _ OPTIONS from the policy.

Fixes: 8b4c3cdd9dd8 (“net: sched: Add policy validation for tc attributes”) Reported-by: Marco Berizzi Signed-off-by: David Ahern Signed-off-by: David S. Miller Signed-off-by: Greg Kroah-Hartman

Display how to add and remove users, set permissions on files, password Add and remove users

You can configure user accounts in Cumulus Linux with read-only or edit permissions for NCLU:

·· For read-only permissions with NCLU add users to the netshow group. Users in the netshow group can run NCLU net show commands, such as net show interface or net show config, and certain general Linux commands, such as ls, cd or man, but cannot run net add, net del or net commit commands.

·· For edit permissions with NCLU add users the netedit group. Users in the netedit group can run NCLU configuration commands, such net add, net del or net commit in addition to NCLU net show commands.

Add a user with show privileges in NCLU.

cu mulus@switch:~$ sudo adduser --ingroup netshow myuser Adding user `myuser’ ... Adding new user `myuser’ (1001) with group `netshow’ …

Assign an existing user to the netshow permissions group.

cu mulus@switch:~$ sudo addgroup myuser netshow Adding user `myuser’ to group `netshow’ ... Adding user myuser to group netshow Done

Set password

https://docs.cumulusnetworks.com/display/DOCS/User+Accounts

Changing the password for your own account.

cumulus@oob-mgmt-server:~$ passwd Changing password for cumulus. (current) UNIX password: Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully

Utilizing sudo to change the password for another user.

cumulus@oob-mgmt-server:~$ sudo passwd cumulus Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully

Set file permissions

Linux permissions dictate 3 things you may do with a file, read, write and execute referred to Linux by a single letter each.

·· (r) read — you may view the contents of the file

·· (w) write — you may change the contents of the file

·· (x) execute — you may execute or run the file if it is a program or script

For every file there are 3 sets of people for whom permissions are specified. The order of permissions is always read, then write, and then execute.

·· owner — a single person who owns the file. (usually file creator, but ownership may be granted to different user)

·· group — every file belongs to a single group.

·· others — everyone else who is not in the group or the owner.

cumulus@oob-mgmt-server:~$ ls -l total 0 drwxr-xr-x 1 cumulus cumulus 96 Feb 21 15:10 cldemo-netq drwx------1 cumulus cumulus 474 Oct 4 12:54 gateone drwxr-xr-x 1 cumulus cumulus 136 Feb 19 18:48 local-git-repo

·· The first character identifies the file type. A dash “-“, is a normal file. “d”, is a directory.

·· Characters 2 through 4 represent permissions for the owner. A letter represents the presence of a permission and a dash “-“, represents the absence of a permission. The owner above all permissions (read, write and execute).

·· Characters 5 through 7 represent permissions for the group. The group above has the ability to read but not write or execute.

·· Characters 8 through 10 represent permissions for others (everyone else). The others above have the execute permission and nothing else.

Chmod has permission arguments that are made up of 3 components.

·· Who are we changing the permission for? [ugoa] — user (or owner), group, others, all

·· Are we granting or revoking the permission — indicated with either a plus ( + ) or minus ( - )

·· Which permission are we setting? — read ( r ), write ( w ) or execute ( x )

cumulus@oob-mgmt-server:~/cldemo-netq$ ls -l total 20 -rw-r--r-- 1 cumulus cumulus 281 Feb 21 15:10 ansible.cfg drwxr-xr-x 1 cumulus cumulus 60 Feb 21 15:10 docker drwxr-xr-x 1 cumulus cumulus 60 Feb 21 15:10 evpn -rw-r--r-- 1 cumulus cumulus 733 Feb 21 15:10 hosts -rw-r--r-- 1 cumulus cumulus 808 Feb 21 15:10 README.md

-rw-r--r-- 1 cumulus cumulus 6445 Feb 21 15:10 setup.yml

cumulus@oob-mgmt-server:~/cldemo-netq$ sudo chmod g+x hosts

cumulus@oob-mgmt-server:~/cldemo-netq$ ls -l total 20 -rw-r--r-- 1 cumulus cumulus 281 Feb 21 15:10 ansible.cfg drwxr-xr-x 1 cumulus cumulus 60 Feb 21 15:10 docker drwxr-xr-x 1 cumulus cumulus 60 Feb 21 15:10 evpn -rw-r-xr-- 1 cumulus cumulus 733 Feb 21 15:10 hosts -rw-r--r-- 1 cumulus cumulus 808 Feb 21 15:10 README.md -rw-r--r-- 1 cumulus cumulus 6445 Feb 21 15:10 setup.yml

Describe the benefits and differences between password login and keybased Password authentication is accomplished with the user prompted for a password upon login, and the password provided by the user is hashed and checked against a file or database. This requires each user to remember their (ideally complex) password and follow proper password security protocols. User forgetting password or locking out their access is quite common and can add administrative burden.

Key based authentication is accomplished by generating a public and private key pair on your jump host(s) or oob-mgmt device(s), and copying the public key file to all of your hosts. This requires the admin to properly generate the key file and copy it successful to 1000s of devices, and is an excellent candidate for automation. This method of login also enables easier automation by simplifying the process for system to system interaction. The authentication lives in a file and is transparent to the user at the time of login, as opposed to the user entering a password. The login mechanism only transits the network once upon initial copy, opposed to each time with password based logins.

https://docs.cumulusnetworks.com/display/DOCS/SSH+for+Remote+Access

Describe the difference between userspace and kernel Linux, memory gets divided into two areas:

·· The user space, which is a set of locations where normal user processes run (everything other than the kernel). The role of the kernel is to manage applications running in this space from messing with each other, and the system.

·· The kernel space, which is the location where the code of the kernel is stored, and executes under.

FIGURE 16

Processes running under the user space have access only to a limited part of memory, whereas the kernel has access to all of the memory. Processes running in user space don’t have access to the kernel space. User space processes can only access a small part of the kernel via an interface exposed by the kernel — the system calls. If a process performs a system call, a software interrupt is sent to the kernel, which then dispatches the appropriate interrupt handler and continues its work after the handler has finished.

This separation is meant to ensure that Linux is as reliable and secure and operating system as possible.

Configure systemd service architecture Display starting, enabling, disabling a service

cumulus@switch:~$ sudo systemctl stop ntp.service

To move a service to run in the management VRF:

1. Stop the service in the default VRF

2. Disable the service from starting automatically in default VRF

3. Start the service in the management VRF

4. Enable the service to start automatically in the management VRF

cumulus@switch:~$ sudo systemctl stop ntp.service

cumulus@switch:~$ sudo systemctl disable ntp.service

cumulus@switch:~$ sudo systemctl start [email protected]

cumulus@switch:~$ sudo systemctl enable [email protected]

BASH overview and purpose https://www.gnu.org/software/bash/

Bash (bourne again shell) is a sh-compatible shell that incorporates useful features from the Korn shell (ksh) and C shell (csh). It is intended to conform to the IEEE POSIX P1003.2/ISO 9945.2 Shell and Tools standard. It gives users interaction to a command line environment to interface with the system and offers functional improvements over sh for both programming and interactive. The improvements offered by Bash include:

·· Command line editing

·· Unlimited size command history

·· Job Control

·· Shell Functions and Aliases

·· Indexed arrays of unlimited size

·· Integer arithmetic in any base from two to sixty-four

Stdin/out/err, utilities, pipes and redirection Every program we run on the command line automatically has three data streams connected to it.

·· STDIN (0) — Standard input (data fed into the program)

·· STDOUT (1) — Standard output (data printed by the program, defaults to the terminal)

·· STDERR (2) — Standard error (for error messages, also defaults to the terminal)

Normally, we will get our output on the screen, which is convenient most of the time, but sometimes we desire to save it into a file to keep as a record, feed into another system, or send it to someone else. The greater than operator “>” indicates to the command line the user wants the programs output (or whatever it sends to STDOUT) to be saved in a file instead of printed to the screen.

If the user redirects to a file which does not exist it will be created automatically, but if redirected into a file which already exists, then the file’s contents will be cleared, and the new output saved to it. Instead the new data can be appended to the file by using the double greater than operator “>>”.

Pipes

Pipes redirect the output of a command to another command. The output of ls -al can be piped to less in order to scroll the output via keypress. The “|” character is used; the command from the example is ls -al | less. The operator feeds the output from the program on the left as input to the program on the right.

The below example feeds the output of net show configuration files into grep to search for directory markers at the start of lines in order to only show the configuration file paths rather than the entire file.

cu mulus@leaf01:~$ net show configuration files | grep ^/ /etc/ntp.conf /etc/timezone /etc/sn m p/sn m pd.conf /etc/frr/daemons /etc/frr/frr.conf /etc/r esolv.c o n f /etc/default/isc-dhcp-relay /etc/default/isc-dhcp-relay6 /etc/default/isc-dhcp-server /etc/default/isc-dhcp-server6 /etc/cumulus/ports.conf /etc/ptp4l.conf /etc/network/interfaces /etc/hostname /etc/hosts /etc/dhcp/dhclient-exit-hooks.d/dhcp-sethostname /etc/vxsnd.conf /usr/lib/python2.7/dist-packages/cumulus/ _ _ chip _ config/mlx/datapath.conf /etc/cumulus/datapath/traffic.conf /etc/hostapd.conf

Display how to change directories

cumulus@leaf01:mgmt-vrf:/$ cd / cumulus@leaf01:mgmt-vrf:/$ pwd / cumulus@leaf01:mgmt-vrf:/$ cd ~ cu mulus@leaf01:m g mt-vrf:~$ pwd /ho m e/cu mulus cumulus@leaf01:mgmt-vrf:~$ cd .. cumulus@leaf01:mgmt-vrf:/home$ pwd /home

Display how to create files

Linux users can create blank files with the commandtouch [options] . They can utilize the redirection methods already covered to create a new file with pertinent information to their current task. Users can copy files with or without manipulation to create new files as well.

cumulus@oob-mgmt-server:~/cldemo-netq$ touch testfile

cumulus@oob-mgmt-server:~/cldemo-netq$ ls ansible.cfg docker evpn hosts README.md setup.yml testfile

cumulus@oob-mgmt-server:~/cldemo-netq$ ls > testfile2

cumulus@oob-mgmt-server:~/cldemo-netq$ cp testfile2 testfile3

cumulus@oob-mgmt-server:~/cldemo-netq$ ls ansible.cfg docker evpn hosts README.md setup.yml testfile testfile2 testfile3

Display how to use sudo

The sudo command allows you to execute a command as superuser or another user as specified by the security policy. Examples of sudo usage are numerous in this document.

https://docs.cumulusnetworks.com/display/DOCS/Using+sudo+to+Delegate+Privileges

Display how to use grep

The grep program can be used for many purposes, such as finding files, searching inside files, including lines before and/or after search item, or for filtering content output for show commands in Cumulus Linux.

cu mulus@leaf01:m g mt-vrf:~$ cat /etc/network/interfaces | grep eth0 auto eth0 iface eth0 inet dhcp

cu mulus@leaf01:m g mt-vrf:~$ cat /etc/network/interfaces | grep -A 2 -B 2 eth0

# Management interface auto eth0 iface eth0 inet dhcp alias management interface vrf mgmt.

cumulus@leaf01:mgmt-vrf:~$ dpkg -l | grep -i cumulus ii cumulus-archive-keyring 4-cl3u5 all GnuPG archive keys for Cumulus apt archives ii cumulus-basefiles 3.7.3 all Cumulus Linux base files for distribution ii cumulus-chassis 0.1-cl3u4 all Cumulus Chassis Daemon ii cumulus-hyperconverged 0.1-cl3u2 all Cumulus Hyperconverged Daemon

ii cumulus-image 3.0-cl3u10 amd64 Cumulus Linux boot manipulation scripts ii cumulus-initramfs 3.1-cl3u4 amd64 Cumulus Linux initramfs integration scripts ii cumulus-net-snmp-addons 1-cl3u7 all Cumulus Networks Enterprise MIBs and pass persist scripts ii cumulus-netq 1.4.0-cl3u10~1537582095.5e7e5e5 all This meta-package provides installation of Cumulus NetQ packages. ii cumulus-platform 3.0-cl3u25 all Cumulus Linux hardware platform common files ii cumulus-snapshot 1.0-cl3u4 amd64 Cumulus Linux btrfs snapshot/ rollback integration scripts ii cumulus-tools 3.7.0-cl3u4 all Cumulus Linux tools and services ii netq-agent 1.4.0-cl3u10~1537996283.40268de all Cumulus NetQ Telemetry Agent for Cumulus Linux ii netq-apps 1.4.0-cl3u10~1537996283.40268de amd64 Cumulus NetQ Fabric Validation Application for Cumulus Linux ii netshow 1.1.7-cl3u11 all Cumulus Linux Network Troublshooting Tool Provider ii onie-tools 3.2-cl3u6 amd64 Cumulus Linux integration scripts for ONIE ii phy-ucode-update 3.0-cl3u3 all Cumulus Linux PHY microcode update utilitiy ii python-cumulus-restapi 0.1-cl3u9 all Rest API for Cumulus Networks ii python-netq-lib 1.4.0-cl3u10~1537996283.40268de all Cumulus NetQ Library for Cumulus Linux ii switchd 1.0-cl3u32 amd64 Cumulus Networks switch chip daemon

Describe file system structure and where files are located

·· / — Everything on a Linux system is located under the/directory, known as the root directory.

·· /bin — contains the essential user binaries (programs) that must be present when the system is mounted in single-user mode. Applications such as Firefox are stored in /usr/bin, while important system programs and utilities such as the bash shell are located in /bin.

·· /boot — contains the files needed to boot the system— for example, the GRUB boot loader’s files and your Linux kernels are stored here. The boot loader’s configuration files aren’t located here, though — they’re in /etc with the other configuration files.

·· /dev — Linux exposes devices as files, and the/dev directory contains a number of special files that represent devices. These are not actual files as we know them, but they appear as files – for example, /dev/sda represents the first SATA drive in the system.

·· /etc — contains configuration files. The/etc directory contains system-wide configuration files.

·· /lib — contains libraries needed by the essential binaries in the /bin and /sbin folder. Libraries needed by the binaries in the /usr/bin folder are located in /usr/lib.

·· /home — contains a home folder for each user which contains the user’s data files and user-specific configuration files. The cumulus user’s home folder is/home/cumulus . Each user only has write permission to their own home folder and must obtain elevated permissions to modify other files on the system.

·· /opt — contains subdirectories for optional software packages. It’s commonly used by proprietary software that doesn’t obey the standard file system hierarchy, which might dump files in /opt/application when installed.

·· /sbin — similar to the /bin directory. It contains essential binaries that are generally intended to be run by the root user for system administration.

·· /usr — contains applications and files used by users, opposed to applications and files used by the system. Non-essential applications are located inside the /usr/bin directory instead of the /bin directory and non-essential system administration binaries are located in the /usr/sbin directory instead of the /sbin directory.

·· /var — is the writable counterpart to the /usr directory, which must be read-only in normal operation. Log files and everything else that would normally be written to /usr during normal operation are written to the /var directory. Log files are one example found in/var/log . Cumulus Linux network configuration files

File Name and Location Explanation Cumulus Linux Documentation

/etc/network/ Network config files, i.e./etc/network/ Switch Port Attributes interfaces & /etc/network/interfaces.d/

/etc/resolv.config DNS resolution Not unique wiki.debian.org/ NetworkConfiguration

/etc/frr/ Routing application responsible for BGP FRRouting Overview and OSPF

/etc/hostname Configuration file for the hostname of Quick Start Guide the switch

/etc/cumulus/acl/* Netfilter configuration Netfilter — ACLs

/etc/cumulus/ports.config Breakout cable configuration file Switch Port Attributes

/etc/cumulus/switchd.config Switchd configuration Configuring Switchd

Additional commonly used files

File Name and Location Explanation Cumulus Linux Documentation

/etc/motd Message of the day wiki.debian.org/motd

/etc/passwd User account information www.debian.org/doc/manuals/ debian-reference/ch04.en.html

/etc/shadow Secure user account information www.debian.org/doc/manuals/ debian-reference/ch04.en.html

/etc/group Defines user group on the switch www.debian.org/doc/manuals/ debian-reference/ch04.en.html

/etc/lldpd.conf Link Layer Discover Protocol (LLDP) Link Layer Discovery Protocol daemon configuration

/etc/lldpd.d/ Configuration directory for LLDP Link Layer Discovery Protocol

/etc/nsswitch.conf Name Service Switch (NSS) TACACS Plus configuration file

/etc/ssh/ SSH configuration files SSH for Remote Access

/etc/sudoers/etc/sudoers.d Best practice is to place changes Using sudo to Delegate Privileges in /etc/sudoers.d/ instead of /etc/sudoers; changes in the /etc/ sudoers.d/ directory are not lost during upgrade. The sudoers filechanged in Cumulus Linux 3.2.

Dynamic Host Configuration Protocol (DHCP) Dynamic Host Configuration Protocol (DHCP) assign IP addresses and other properties such as DNS, default gateways, provisioning or boot information to end points dynamically in order to increase efficiency in operation and administrative resources. Devices running Linux can function as a DHCP server or DHCP client.

Server configuration file example:

default-lease-time 600; max-lease-time 7200;

subnet 10.1.1.0 netmask 255.255.255.0 {

range 10.1.1.3 10.1.1.254; option domain-name-servers 10.1.1.1, 8.8.8.8; option routers 10.1.1.1; }

subnet 192.168.0.0 netmask 255.255.0.0 { }

host printer { hardware Ethernet 00:01:da:b4:3e:45; fi xed-add ress 10.1.1.100; }

host web-server { hardware Ethernet 00:02:a9:df:31:90; fi xed-add ress 10.1.1.101; }

Common DHCP client configuration for eth0 interface in/etc/network/ interfaces:

auto eth0 iface eth0 inet dhcp

The dhclient program can be run to interact with DHCP manually on an interface. A release and renew is exampled below.

cumulus@oob-mgmt-server:~/cldemo-netq$ sudo dhclient -r eth0

cumulus@oob-mgmt-server:~/cldemo-netq$ sudo dhclient eth0

Overlay routing concepts

Describe and configure a VXLAN VXLAN overview

Virtual Extensible LAN is a standards-based overlay technology defined in RFC7348 designed to provide layer 2 adjacency over a layer 3 network via encapsulation and decapsulation through a component called a VTEP (VxLAN Tunnel End Point). A VTEP has an IP address in the underlay network and also has one or more VNI’s associated. When frames from one of these VNI’s arrives at the ingress VTEP, the VTEP encapsulates it with UDP and IP headers.

The encapsulated packet is sent over the IP network to the Egress VTEP. When it arrives, the VTEP removes the IP and UDP headers, and delivers the frame as normal.

VXLAN configuration

cumulus@leaf01:~$ net add loopback lo ip address 10.0.0.11/32 cumulus@leaf01:~$ net add vxlan vni-10 vxlan id 10 cumulus@leaf01:~$ net add vxlan vni-10 vxlan local-tunnelip 10.0.0.11 cumulus@leaf01:~$ net add vxlan vni-10 vxlan remoteip 10.0.0.12 cumulus@leaf01:~$ net add vxlan vni-10 vxlan remoteip 10.0.0.13 cumulus@leaf01:~$ net add vxlan vni-10 vxlan remoteip 10.0.0.14 cumulus@leaf01:~$ net add vxlan vni-10 bridge access 10 cumulus@leaf01:~$ net pending cumulus@leaf01:~$ net commit

Describe the difference between asymmetric and symmetric routing Asymmetric routing

FIGURE 17

In distributed asymmetric routing, each VTEP acts as a layer 3 gateway, performing routing for its attached hosts. The routing is called asymmetric because only the ingress VTEP performs routing, the egress VTEP only performs the bridging. Traffic always travels on different VNIs; always the destination VNI. Asymmetric routing is easy to deploy as it can be achieved with only host routing and does not involve any interconnecting VNIs. However, each VTEP must be provisioned with all VLANs/VNIs — the subnets between which communication can take place; this is required even if there are no locally-attached hosts for a particular VLAN.

The only additional configuration required to implement asymmetric routing beyond the standard configuration for a layer 2 VTEP is to ensure each VTEP has all VLANs (and corresponding VNIs) provisioned on it and the SVI for each such VLAN is configured with an Anycast gateway IP/MAC address.

Use the asymmetric model if:

·· You plan to deploy all VNIs on all leaf switches anyway

·· You have a small number of VNIs in your data center or POD

·· Your data center can be broken into PODs, with the VLANs/subnets contained in each POD Symmetric routing

In distributed symmetric routing, each VTEP acts as a layer 3 gateway, performing routing for its attached hosts. This is the same as in asymmetric routing. The difference with symmetric routing is, both the ingress VTEP and egress VTEP route the packets. Therefore, it can be compared to the traditional routing behavior of routing to a next hop router. In the VXLAN encapsulated packet, the inner destination MAC address is set to the router MAC address of the egress VTEP as an indication that the egress VTEP is the next hop and also needs to perform routing. All routing happens in the context of a tenant (VRF).

For a packet received by the ingress VTEP from a locally attached host, the SVI interface corresponding to the VLAN determines the VRF. For a packet received by the egress VTEP over the VXLAN tunnel, the VNI in the packet has to specify the VRF. For symmetric routing, this is a VNI corresponding to the tenant and is different from either the source VNI or the destination VNI. This VNI is referred to as the layer 3 VNI, transit VNI, or interconnecting VNI; it has to be provisioned by the operator and is exchanged through the EVPN control plane. In order to make the distinction clear, the regular VNI, which is used to map a VLAN, is referred to as the layer 2 VNI.

·· There is a one-to-one mapping between a layer 3 VNI and a tenant (VRF).

·· The VRF to layer 3 VNI mapping has to be consistent across all VTEPs. The layer 3 VNI has to be provisioned by the operator.

·· Layer 3 VNI and layer 2 VNI cannot share the same number space.

·· In an MLAG configuration, the SVI used for the layer 3 VNI cannot be part of the bridge. This ensures that traffic tagged with that VLAN ID is not forwarded on the peer link or other trunks.

FIGURE 18

FIGURE 19

For EVPN symmetric routing, additional configuration is required:

·· Configure a per-tenant VXLAN interface that specifies the layer 3 VNI for the tenant. This VXLAN interface is part of the bridge and router MAC addresses of remote VTEPs is installed over this interface.

·· Configure an SVI (layer 3 interface) corresponding to the per-tenant VXLAN interface. This is attached to the tenant’s VRF. Remote host routes for symmetric routing are installed over this SVI.

·· Specify the mapping of VRF to layer 3 VNI. This configuration is for the BGP control plane.

Use the symmetric model if:

·· Your VLANs/subnets/VNIs are widely dispersed

·· Your VLANs/subnets/VNIs are provisioned on the fly

·· You need external routing with Cumulus Linux 3.5

Describe the basics of EVPN, a BGP EVPN control plane, and the different route types Ethernet Virtual Private Network (EVPN)

VXLAN is the de facto technology for implementing network virtualization in the data center, enabling layer 2 segments to be extended over an IP core (the underlay). The initial definition of VXLAN (RFC 7348) did not include any control plane and relied on a flood-and-learn approach for MAC address. An alternate deployment model was to use a controller or a technology such as Lightweight Network Virtualization (LNV) in Cumulus Linux (you cannot use EVPN and LNV at the same time).

Ethernet Virtual Private Network (EVPN) is a standards-based control plane for VXLAN defined in RFC 7432 and draft-ietf-bess-evpn-overlay that allows for building and deploying VXLANs at scale. It relies on multi-protocol BGP (MP-BGP) for exchanging information and is based on BGP-MPLS IP VPNs (RFC 4364). It has provisions to enable not only bridging between end systems in the same layer 2 segment but also routing between different segments (subnets). There is also inherent support for multi-tenancy. EVPN is often referred to as the means of implementing controller-less VXLAN.

BGP EVPN control plane

EVPN address-family is supported with both eBGP and iBGP peering. If the underlay routing is provisioned using eBGP, the same eBGP session can also be used to carry EVPN routes. For example, in a typical 2-tier Clos network topology where the leaf switches are the VTEPs, if eBGP sessions are in use between the leaf and spine switches for the underlay routing, the same sessions can be used to exchange EVPN routes and the spine switches merely act as “route forwarders” as they do not install any forwarding state as they are not VTEPs. When EVPN routes are exchanged over iBGP peering, OSPF can be used as the IGP or the next hops can also be resolved using iBGP.

Key features of Cumulus Linux regarding EVPN as the control plane for VXLAN:

·· VNI membership exchange between VTEPs using EVPN type-3 (Inclusive multicast Ethernet tag) routes.

·· Exchange of host MAC and IP addresses using EVPN type-2 (MAC/IP advertisement) routes.

·· Support for host/VM mobility (MAC and IP moves) through exchange of the MAC Mobility Extended community.

·· Dual-attached host support via VXLAN active-active mode. MAC synchronization between peers uses MLAG.

·· Support for ARP/ND suppression, providing VTEPs with the ability to suppress ARP flooding over VXLAN tunnels.

·· Support for exchange of static (sticky) MAC addresses through EVPN.

·· Support for distributed symmetric routing between different subnets.

·· Support for distributed asymmetric routing between different subnets.

·· Support for centralized routing.

·· Support for prefix-based routing using EVPN type-5 routes (EVPN IP prefix route)

·· Support for layer 3 multi-tenancy.

·· Support for IPv6 tenant routing.

·· Symmetric routing, asymmetric routing and prefix-based routing are supported for IPv4/IPv6 hosts and prefixes.

·· ECMP support for overlay networks on RIOT-capable Broadcom switches (Trident 3, Maverick, Trident 2+) in addition to Mellanox Spectrum-A1 and Tomahawk switches.

EVPN route types

Type Route Description RFC Description

0 Reserved 7432

1 Ethernet Auto 7432

Discovery (A-D)

2 Mac/IP Adverstisement 7432 Type 2 is MAC address advertisements. Each VTEP advertises any MAC address learned in its MAC and IP Neighbor table into BGP as a type 2 EVPN advertisement. Type 2 routes are generally used for communication within a data center.

3 Inclusive Multicast 7432 Type 3 is for VTEP discovery.

4 Ethernet Segment 7432 Type 5 is an IP Prefix advertisement used with distributed architecture to advertise external, inter-data center and/or inter-POD routes to all local leafs within a VRF and can only be used with an L3VNI. A type 5 route may also be used for inter-POD or interdata center routing.

5 IP Prefix Route IETF Draft

6 Selective Multicast Ethernet Tag IETF Draft

7 IGMP Join Synch IETF Draft

8 IGMP Leave Synch IETF Draft

9 Per-region I-PMSI A-D IETF Draft

10 S-PMSI A-D IETF Draft

11 Leaf A-D IETF Draft

12-255 Unassigned

The following components are part of a BGP EVPN type 2 route

·· Route Distinguisher

·· Ethernet Segment Identifier

·· Ethernet Tag ID

·· MAC Address Length

·· MAC Address

·· IP Address Length

·· IP Address

·· Label 1 (L2VNI)

·· Label 2 (L3VNI)

A type 2 route is used for:

·· All VLAN/VXLAN bridging

·· Intra-data center or intra-POD VXLAN routing

·· Inter data center routing with stretched VXLAN tunnels (VXLAN tunnels between data centers or PODs)

If external layer 3 connectivity is required, a separate route type, type 5, is used. BGP EVPN type 5 route components:

·· Route Distinguisher

·· Ethernet Segment Identifier

·· Ethernet Tag ID

·· IP Prefix Length

·· IP Prefix

·· GW IP address

·· Label (L3VNI)

A type 5 EVPN route is used for:

·· All traffic destined to the Internet

·· Inter-data center or inter-POD VXLAN routing

FIGURE 20

Core Cumulus concepts

Describe awareness & interaction between NCLU & ifupdown2 NCLU provides an easy to use wrapper and guard rails to manage interfaces in Cumulus Linux using ifupdown2.

Cumulus Networks implemented the ifreload feature in ifupdown2, the network interface configuration utility. ifupdown2 interacts with the /etc/network/interfaces flat file, which controls all network configuration (VLANs, MTU, IP addressing) except routing. When the/etc/network/interfaces file is edited and overwritten, issuing an ifreload -a or systemctl reload networking.service causes ifupdown2 to apply changes only to the part of the configuration that was modified.

FIGURE 21

Configure interfaces https://docs.cumulusnetworks.com/display/DOCS/Interface+Configuration+and+Management

An interface can be placed into an admin down state. The interface remains down after any future reboots or applying configuration changes withifreload -a.

cumulus@switch:~$ net add interface swp1 link down

Port lists or ranges can be specified using the commas and dashes.

cu mulus@leaf01:m g mt-vrf:~$ net add interface swp1-2 link down

cumulus@leaf01:mgmt-vrf:~$ net show interface all | grep swp[1-2] ADMDN swp1 N/A 9000 BondMember Master: bond01(DN) ADMDN swp2 N/A 9000 BondMember Master: bond02(DN) UP swp51 1G 9216 NotConfigured spine01 (swp1) UP swp52 1G 9216 NotConfigured spine02 (swp1) bond01 Bond Members: swp1(ADMDN) bond02 Bond Members: swp2(ADMDN)

cumulus@leaf01:mgmt-vrf:~$ net show interface swp1-2 Name MAC Speed MTU Mode ------ADMDN swp1 44:38:39:00:02:05 N/A 9000 BondMember

Alias ------to Server01

Routing ------Interface swp1 is down Link ups: 1 last: 2019/02/28 14:17:11.67 Link downs: 1 last: 2019/02/28 15:18:23.17 PTM status: disabled vrf: default Description: to Server01 index 7 metric 0 mtu 9000 speed 1000 flags: Type: Ethernet

HWaddr: 44:38:39:00:02:05 Interface Type Other

Name MAC Speed MTU Mode ------ADMDN swp2 44:38:39:00:02:06 N/A 9000 BondMember

Alias ------to Server02

Routing ------Interface swp2 is down Link ups: 1 last: 2019/02/28 14:17:11.67 Link downs: 1 last: 2019/02/28 15:18:23.25 PTM status: disabled vrf: default Description: to Server02 index 8 metric 0 mtu 9000 speed 1000 flags: Type: Ethernet HWaddr: 44:38:39:00:02:06 Interface Type Other

IPv4 and IPv6 addresses can be added to interfaces in the following manner.

cumulus@switch:~$ net add interface swp1 ip address 12.0.0.1/30 cumulus@switch:~$ net add interface swp1 ip address 12.0.0.2/30 cumulus@switch:~$ net add interface swp1 ipv6 address 2001:DB8::1/126

Interface Descriptions can be added by creating an alias, which will be displayed in output and SNMP OID IF-MIB::ifAlias. The alias can be up to 255 characters.

cu mulus@switch:~$ net add interface swp1 alias hypervisor _ port _ 1

cumulus@switch$ net show interface swp1 Name MAC Speed MTU Mode ------UP swp1 44:38:39:00:00:04 1G 1500 Access/L2 Alias

----- hypervisor _ port _ 1

cumulus@switch:~$ net show interface alias State Name Mode Alias ------UP bond01 LACP UP eth0 Mgmt UP lo Loopback loopback interface UP mgmt Interface/L3 UP swp1 BondMember hypervisor _ port _ 1 UP swp2 BondMember to Server02

Create a topology file and verify cabling with PTM https://docs.cumulusnetworks.com/display/DOCS/Prescriptive+Topology+Manager+-+PTM

PTM overview

In data center topologies, ensuring proper cabling can be a time-consuming and error-prone endeavor. Prescriptive Topology Manager (PTM) is a dynamic cabling verification tool to help detect and eliminate such errors. It takes a Graphviz-DOT specified network cabling plan, stored in a topology.dot file, and couples it with runtime information derived from LLDP to verify that the cabling matches the specification. The check is performed on every link transition on each node in the network.

FIGURE 22

The topology.dot file can be customized to control ptmd at both the global/network level and the node/port level. PTM runs as a daemon, named ptmd. For more information, see man ptmd(8).

The same topology.dot file should be used on all switches, and DO NOT split the file per device, as this allows for easier automation by pushing/pulling the same exact file on each device.

Host-only parameters apply to the entire host on which PTM is running. You can include the hostnametype host-only parameter, which specifies whether PTM should use only the host name (hostname) or the fully- qualified domain name (fqdn) while looking for the self-node in the graph file.

Global parameters apply to every port listed in the topology file. There are two global parameters: LLDP and BFD. LLDP is enabled by default; if no keyword is present, default values are used for all ports. However, BFD is disabled if no keyword is present, unless there is a per-port override configured.

Per-port parameters provide finer-grained control at the port level. These parameters override any global or compiled defaults.

Basic DOT example

graph G { “spine1”:”swp1” -- “leaf1”:”swp1”; “spine1”:”swp2” -- “leaf2”:”swp1”; “spine2”:”swp1” -- “leaf1”:”swp2”; “spine2”:”swp2” -- “leaf2”:”swp2”; “leaf1”:”swp3” -- “leaf2”:”swp3”; “leaf1”:”swp4” -- “leaf2”:”swp4”; “leaf1”:”swp5s0” -- “server1”:”eth1”; “leaf2”:”swp5s0” -- “server2”:”eth1”; }

PTM templates

Templates provide flexibility in choosing different parameter combinations and applying them to a given port. A template instructs ptmd to reference a named parameter string instead of a default one. There are two parameter strings ptmd supports:

·· bfdtmpl, which specifies a custom parameter tuple for BFD

·· lldptmpl, which specifies a custom parameter tuple for LLDP

In this template, LLDP1 and LLDP2 are templates for LLDP parameters while BFD1 and BFD2 are templates for BFD parameters. The templates are then referenced in the connectivity entries.

graph G { LLDP=”” BFD=”upMinTx=300,requiredMinRx=100” BFD1=”upMinTx=200,requiredMinRx=200” BFD2=”upMinTx=100,requiredMinRx=300” LLDP1=”match _ type=ifname” LLDP2=”match _ type=portdescr” “cumulus”:”swp44” -- “qct-ly2-04”:”swp20” [BFD=”bfdtmpl=BFD1”, LLDP=”lldptmpl=LLDP1”]

“cumulus”:”swp46” -- “qct-ly2-04”:”swp22” [BFD=”bfdtmpl=BFD2”, LLDP=”lldptmpl=LLDP2”] “cumulus”:”swp46” -- “qct-ly2-04”:”swp22” }

Configure, describe, and troubleshoot BGP unnumbered operation BGP unnumbered overview

https://docs.cumulusnetworks.com/display/DOCS/Border+Gateway+Protocol+-+BGP#BorderGateway Protocol-BGP-unnumberedBGPUnnumberedInterfaces

BGP unnumbered enables the peering of devices without unique interface IP addresses. In BGP, configure unnumbered interfaces using extended next hop encoding (ENHE), which is defined by RFC 5549. BGP unnumbered interfaces provide a means of advertising an IPv4 route with an IPv6 next hop. Prior to RFC 5549, an IPv4 route could be advertised only with an IPv4 next hop.

FIGURE 23

BGP unnumbered interfaces are particularly useful in deployments where IPv4 prefixes are advertised through BGP over a section without any IPv4 address configuration on links. This results in vast IP space and administrative resources saved, while making automation significantly easier.

Every router or end host must have an IPv4 address to complete a traceroute of IPv4 addresses. In this case, the IPv4 address used is that of the loopback device. Even if ENHE is not used in the data center, link addresses are not typically advertised. Assigning an IP address to the loopback device is essential.

·· Link addresses take up valuable FIB resources, and the number of such addresses can be quite large, increasing quickly based on number of spines, leafs and port density

·· 4 Spines * 32 interfaces * 2 IPs = 256 IP addresses or 8 * 96 * 2 = 1536 IP Addresses

·· Link addresses expose an additional attack vector for intruders to use to either break in or engage in DDOS attacks

BGP unnumbered functional summary:

·· BGP unnumbered uses the interface’s IPv6 LLA to set up a BGP session with a peer

·· The IPv6 LLA of the remote end is discovered via IPv6’s Router Advertisement (RA) protocol

·· RA provides not only the remote end’s LLA, but also its corresponding MAC address

·· BGP uses RFC 5549 to encode IPv4 routes as reachable over an IPv6 nexthop, using the IPv6 LLA as the nexthop

·· The RIB process programs a static ARP entry with a reserved IPv4 LLA, 169.254.0.1, with the MAC address set to the one learned via RA

·· BGP hands down to the RIB process, IPv4 routes with the IPv6 LLA as the nexthop

·· The RIB process converts the nexthop to 169.254.0.1 and the outgoing interface before programming the route in the forwarding table

BGP unnumbered configuration

To configure a BGP unnumbered interface, IPv6 neighbor discovery router advertisements must be enabled. The interval specified is measured in seconds and defaults to 10.

In Cumulus Linux 3.7.1 and earlier, ENHE is sent only for the link-local address peering. In Cumulus Linux 3.7.2 and later, extended next hop encoding can be sent for the both link-local and global unicast address peering.

cumulus@switch:~$ net add bgp autonomous-system 65020 cumulus@switch:~$ net add bgp router-id 10.0.0.21 cumulus@switch:~$ net add bgp bestpath as-path multipath-relax cumulus@switch:~$ net add bgp bestpath compare-routerid cumulus@switch:~$ net add bgp neighbor fabric peer-group cumulus@switch:~$ net add bgp neighbor fabric remote-as external cumulus@switch:~$ net add bgp neighbor fabric description Internal Fabric Network cumulus@switch:~$ net add bgp neighbor fabric capability extended-nexthop cumulus@switch:~$ net add bgp neighbor swp1 interface peer-group fabric cumulus@switch:~$ net add bgp neighbor swp2 interface peer-group fabric cumulus@switch:~$ net add bgp neighbor swp3 interface peer-group fabric cumulus@switch:~$ net add bgp neighbor swp4 interface peer-group fabric cumulus@switch:~$ net add bgp neighbor swp29 interface peer-group fabric cumulus@switch:~$ net add bgp neighbor swp30 interface peer-group fabric

BGP unnumbered troubleshooting

BGP unnumbered troubleshooting is not that different, except that you need to focus on the IPv6 addresses and switchport information as the next hop information.

Quick snapshot of neighbor information and overall summary is useful for most issues.

cu mulus@leaf01:m g mt-vrf:~$ net show bgp summary show bgp ipv4 unicast summary ======BGP router identifier 10.0.0.11, local AS number 65011 vrf-id 0 BGP table version 12 RIB entries 15, using 2280 bytes of memory Peers 2, using 39 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd spine01(swp51) 4 65020 5608 5666 0 0 0 04:36:06 6 spine02(swp52) 4 65020 5608 5667 0 0 0 04:36:05 6

Total number of neighbors 2

show bgp ipv6 unicast summary ======% No BGP neighbors found

show bgp l2vpn evpn summary ======BGP router identifier 10.0.0.11, local AS number 65011 vrf-id 0 BGP table version 0 RIB entries 15, using 2280 bytes of memory Peers 2, using 39 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd spine01(swp51) 4 65020 5608 5666 0 0 0 04:36:06 32 spine02(swp52) 4 65020 5608 5667 0 0 0 04:36:05 32

Total number of neighbors 2

We also want to observe and understand the BGP prefix information.

cu mulus@leaf01:m g mt-vrf:~$ net show bgp show bgp ipv4 unicast ======BGP table version is 12, local router ID is 10.0.0.11 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed

Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path *> 10.0.0.11/32 0.0.0.0 0 32768 i *= 10.0.0.12/32 swp52 0 65020 65012 i *> swp51 0 65020 65012 i *= 10.0.0.13/32 swp52 0 65020 65013 i *> swp51 0 65020 65013 i *= 10.0.0.14/32 swp52 0 65020 65014 i *> swp51 0 65020 65014 i *> 10.0.0.21/32 swp51 0 0 65020 i *> 10.0.0.22/32 swp52 0 0 65020 i * 10.0.0.112/32 swp52 0 65020 65012 i * swp51 0 65020 65012 i *> 0.0.0.0 0 32768 i *> 10.0.0.134/32 swp51 0 65020 65013 i *= swp52 0 65020 65013 i

Displayed 8 routes and 14 total paths

show bgp ipv6 unicast ======No BGP prefixes displayed, 0 exist

FRRouting RIB command output.

cu mulus@leaf01:m g mt-vrf:~$ net show route ipv4 Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, > - selected route, * - FIB route

C>* 10.0.0.11/32 is directly connected, lo, 04:38:48 B>* 10.0.0.12/32 [20/0] via fe80::4638:39ff:fe00:601, swp51, 04:38:41 * via fe80::4638:39ff:fe00:701, swp52, 04:38:41 B>* 10.0.0.13/32 [20/0] via fe80::4638:39ff:fe00:601, swp51, 04:38:41 * via fe80::4638:39ff:fe00:701, swp52, 04:38:41 B>* 10.0.0.14/32 [20/0] via fe80::4638:39ff:fe00:601, swp51, 04:38:30 * via fe80::4638:39ff:fe00:701, swp52, 04:38:30 B>* 10.0.0.21/32 [20/0] via fe80::4638:39ff:fe00:601, swp51, 04:38:42 B>* 10.0.0.22/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 04:38:41

C>* 10.0.0.112/32 is directly connected, lo, 04:38:21 B>* 10.0.0.134/32 [20/0] via fe80::4638:39ff:fe00:601, swp51, 04:38:25 * via fe80::4638:39ff:fe00:701, swp52, 04:38:25 C * 10.1.3.0/24 is directly connected, vlan13-v0, 04:38:23 C>* 10.1.3.0/24 is directly connected, vlan13, 04:38:23 C * 10.2.4.0/24 is directly connected, vlan24-v0, 04:38:23 C>* 10.2.4.0/24 is directly connected, vlan24, 04:38:23 C>* 169.254.1.0/30 is directly connected, peerlink.4094, 04:38:42

The following command shows how the IPv4 link-local address 169.254.0.1 is used to install the route and static neighbor entry to facilitate proper forwarding without having to install an IPv4 prefix with IPv6 next hop in the kernel.

cu mulus@leaf01:m g mt-vrf:~$ net show route 10.0.0.12 RIB entry for 10.0.0.12 ======Routing entry for 10.0.0.12/32 Known via “bgp”, distance 20, metric 0, best Last update 04:24:52 ago * fe80::4638:39ff:fe00:601, via swp51 * fe80::4638:39ff:fe00:701, via swp52

FIB entry for 10.0.0.12 ======10.0.0.12 proto bgp metric 20 nexthop via 169.254.0.1 dev swp51 weight 1 onlink nexthop via 169.254.0.1 dev swp52 weight 1 onlink

This can be compared to the BGP routing information.

cu mulus@leaf01:m g mt-vrf:~$ n et sho w b g p 10.0.0.12 BGP routing table entry for 10.0.0.12/32 Paths: (2 available, best #2, table default) Advertised to non peer-group peers: spine01(swp51) spine02(swp52) 65020 65012 fe80::4638:39ff:fe00:701 from spine02(swp52) (10.0.0.22) (fe80::4638:39ff:fe00:701) (used) Origin IGP, valid, external, multipath AddPath ID: RX 0, TX 9 Last update: Thu Feb 28 14:16:52 2019

65020 65012 fe80::4638:39ff:fe00:601 from spine01(swp51) (10.0.0.21) (fe80::4638:39ff:fe00:601) (used) Origin IGP, valid, external, multipath, bestpath-from-AS 65020, best AddPath ID: RX 0, TX 4 Last update: Thu Feb 28 14:16:51 2019

A more detailed view of a BGP Neighbor can be checked for neighbor related issues.

cu mulus@leaf01:m g mt-vrf:~$ net show bgp neighbor swp51 BGP neighbor on swp51: fe80::4638:39ff:fe00:601, remote AS 65020, local AS 65011, external link Hostname: spine01 BGP version 4, remote router ID 10.0.0.21 BGP state = Established, up for 04:39:14 Last read 00:00:00, Last write 00:00:01 Hold time is 9, keepalive interval is 3 seconds Neighbor capabilities: 4 Byte AS: advertised and received AddPath: IPv4 Unicast: RX advertised IPv4 Unicast and received L2VPN EVPN: RX advertised L2VPN EVPN and received Extended nexthop: advertised and received Address families by peer: IPv4 Unicast Route refresh: advertised and received(old & new) Address Family IPv4 Unicast: advertised and received Address Family L2VPN EVPN: advertised and received Hostname Capability: advertised (name: leaf01,domain name: n/a) received (name: spine01,domain name: n/a) Graceful Restart Capabilty: advertised and received Remote Restart timer is 120 seconds Address families by peer: none Graceful restart informations: End-of-RIB send: IPv4 Unicast, L2VPN EVPN End-of-RIB received: IPv4 Unicast, L2VPN EVPN Message statistics: Inq depth is 0 Outq depth is 0 Sent Rcvd Opens: 1 1

Notifications: 0 0 Updates: 143 85 Keepalives: 5585 5585 Route Refresh: 0 0 Capability: 0 0 Total: 5729 5671 Minimum time between advertisement runs is 0 seconds

For address family: IPv4 Unicast Update group 1, subgroup 1 Packet Queue length 0 Community attribute sent to this neighbor(all) 6 accepted prefixes

For address family: L2VPN EVPN Update group 2, subgroup 2 Packet Queue length 0 NEXT _ HOP is propagated unchanged to this neighbor Community attribute sent to this neighbor(all) advertise-all-vni 32 accepted prefixes

Connections established 1; dropped 0 Last reset never Local host: fe80::4638:39ff:fe00:201, Local port: 39933 Foreign host: fe80::4638:39ff:fe00:601, Foreign port: 179 Ne xt ho p: 10.0.0.11 Nexthop global: fe80::4638:39ff:fe00:201 Nexthop local: fe80::4638:39ff:fe00:201 BGP connection: shared network BGP Connect Retry Timer in Seconds: 10 Estimated round trip time: 2 ms Read thread: on Write thread: on

cu mulus@leaf01:m g mt-vrf:~$ net show bgp ipv4 unicast 10.0.0.12 BGP routing table entry for 10.0.0.12/32 Paths: (2 available, best #2, table default) Advertised to non peer-group peers: spine01(swp51) spine02(swp52) 65020 65012 fe80::4638:39ff:fe00:701 from spine02(swp52) (10.0.0.22) (fe80::4638:39ff:fe00:701) (used)

Origin IGP, valid, external, multipath AddPath ID: RX 0, TX 9 Last update: Thu Feb 28 14:16:51 2019

Link-local addresses validation

Verify that the device has learned the IPv6 link-local neighbor ip.

cu mulus@leaf01:m g mt-vrf:~$ net show interface swp51 Name MAC Speed MTU Mode ------UP swp51 44:38:39:00:02:01 1G 9216 NotConfigured

Alias ----- to Spine01

cl-netstat counters ------RX _ OK RX _ ERR RX _ DRP RX _ OVR TX _ OK TX _ ERR TX _ DRP TX _ OVR ------25507 0 2 0 13074 0 0 0

LLDP Details ------LocalPort RemotePort(RemoteHost) ------swp51 swp1(spine01)

Routing ------Interface swp51 is up, line protocol is up Link ups: 0 last: (never) Link downs: 0 last: (never) PTM status: disabled

vrf: default Description: to Spine01 index 3 metric 0 mtu 9216 speed 1000 flags: Type: Ethernet HWaddr: 44:38:39:00:02:01 inet6 fe80::4638:39ff:fe00:201/64 Interface Type Other ND advertised reachable time is 0 milliseconds ND advertised retransmit interval is 0 milliseconds ND router advertisements sent: 1608 rcvd: 1603 ND router advertisements are sent every 10 seconds ND router advertisements lifetime tracks ra-interval ND router advertisement default router preference is medium Hosts use stateless autoconfig for addresses. Neighbor address(s): inet6 fe80::4638:39ff:fe00:601/128

Describe how to manage FRR

By default, FRR configuration settings are stored in an integrated file containing all routing protocols, /etc/frr/frr.conf. If this is disabled each routing protocol process saves their configuration in a different file.

FIGURE 24

NCLU can manage configurations and interact with FRR, as shown throughout this document and specifically in the previous BGP unnumbered section.

The FRR interactive command line can be accessed with the command sudo vtysh. This command line interface feels familiar to traditional networking vendors with question mark completion. FRRouting inherits the IP addresses and associated routing tables for the network interfaces from the /etc/network/interfaces file, and this is the recommended way to define the addresses. Do NOT create interfaces using FRRouting.

Static routes added via FRRouting can be deleted via Linux shell, but while possible, should be avoided. Routes added by FRRouting should only be deleted by FRRouting, otherwise FRRouting might not be able to clean its internal state completely and incorrect routing could occur.

FRR can be manually enabled by editing the file/etc/frr/daemons , and then enabling and starting the FRRouting service.

cu mulus@leaf01:m g mt-vrf:~$ cat /etc/frr/daemons zebra=yes bgpd=yes ospfd=no ospf6d=no ripd=no ripngd=no isisd=no pimd=no ldpd=no nhrpd=no eigrpd=no babeld=no sharpd=no pbrd=no

cumulus@switch:~$ sudo systemctl enable frr.service

cumulus@switch:~$ sudo systemctl start frr.service

The default FRR configuration can be applied by deleting the/etc/frr/frr.conf file and restarting the service.

cumulus@switch:~$ sudo rm /etc/frr/frr.conf

cumulus@switch:~$ sudo systemctl restart frr.service

During configuration updates FRR is reloaded for changes to take effect and applies only the changes made and synchronizes state with the configuration in/etc/frr/frr.conf . This option is not available when using a non-integrated configuration and separate protocol configuration files.

cumulus@switch:~$ sudo systemctl reload frr.service

Describe NCLU and display how to leverage help, add/remove config NCLU overview

https://docs.cumulusnetworks.com/display/DOCS/Network+Command+Line+Utility+-+NCLU

The Network Command Line Utility (NCLU) is a command line interface for Cumulus Networks products simplifying the networking configuration process for all users. NCLU resides in the Linux user space and provides consistent access to networking commands directly through bash, making configuration and troubleshooting simple and easy; no need to edit files or enter modes and sub-modes. NCLU provides these benefits:

·· Embeds help, examples, and automatic command checking with suggestions in case you enter a typo

·· Runs directly from and integrates with bash, while being interoperable with the regular way of accessing underlying configuration files and automation

·· Configures dependent features automatically so that you don’t have to

·· Every configuration change with NCLU is saved in a snapshot

The NCLU wrapper utility “net” is capable of configuring layer 2 and layer 3 features of the networking stack, installing ACLs and VXLANs, rolling back and deleting snapshots, as well as providing monitoring and troubleshooting functionality for these features. You can configure both the/etc/network/interfaces and /etc/frr/frr.conf files with net, in addition to running show and clear commands related to ifupdown2 and FRRouting.

Use the following workflow to stage and commit changes to Cumulus Linux with NCLU:

1. Use the net add and net del commands to stage and remove configuration changes

2. Use the net pending command to review staged changes

3. Use net commit and net abort to commit and delete staged changes

The net commit command applies the changes to the relevant configuration files, such as /etc/network/interfaces, then runs necessary follow on commands to enable the configuration, such as ifreload -a. If two different users try to commit a change at the same time, NCLU displays a warning but implements the change according to the first commit received, and the second user will need to abort their commit.

·· net show is a series of commands to view various parts of the network configuration. Usenet show configuration to view the entire network configuration,net show commit history for a history of commits by NCLU.

·· net clear provides a way to clear net show counters, BGP and OSPF neighbor content, and more.

·· net rollback provides a mechanism to revert back to an earlier configuration.

·· net commit confirm requires the user to press “enter” to commit changes using NCLU within 10 seconds, or the commit automatically reverts and no changes are made.

·· net commit description enables you to provide a descriptive summary of the changes you are about to commit.

·· net commit permanent retains the snapshot taken when committing the change. Otherwise, the snapshots created from NCLU commands are cleaned up periodically with a snapper cron job.

·· net commit delete deletes one or more snapshots created when committing changes with NCLU.

·· net del all deletes all configurations and stops the IEEE 802.1X service. NCLU help

NCLU offers tab completion for help when using individual commands, as well as offering a specific help command option which will search for commands including that keyword. If the keywords returns no results you will receive an error. The basic form of the command returns comprehensive information to guide the user.

cu mulus@leaf01:m g mt-vrf:~$ net help swp1 ERROR: There are no commands with keyword(s) ‘swp1’

cu mulus@leaf01:m g mt-vrf:~$ net help clag peer The following commands contain keyword(s) ‘clag’, ‘peer’

net add clag peer sys-mac interface (primary|secondary) backup-ip vrf net add interface clag peer-ip (||linklocal) net del clag peer net del interface clag peer-ip [||linklocal] net show clag peer-lacp-rate

cu mulus@switch:~$ net help

Usage: # net [] [help] # # net is a command line utility for networking on Cumulus Linux switches. # # COMMANDS are listed below and have context specific arguments which can # be explored by typing “” or “help” anytime while using net. # # Use ‘man net’ for a more comprehensive overview.

net abort net commit [verbose] [confirm] [description ] net commit delete (|) net help [verbose] net pending net rollback (|last)

net show commit (history|||last) net show rollback (|last) net show configuration [commands|files|acl|bgp|ospf|ospf6|interface ]

Options:

# Help commands help : context sensitive information; see section below example : detailed examples of common workflows

# Configuration commands add : add/modify configuration del : remove configuration

# Commit buffer commands abort : abandon changes in the commit buffer commit : apply the commit buffer to the system pending : show changes staged in the commit buffer rollback : revert to a previous configuration state

# Status commands show : show command output clear : clear counters, BGP neighbors, etc

cu mulus@switch:~$ net help bestpath The following commands contain keyword(s) ‘bestpath’

NCLU built in examples

NCLU has a number of built in examples to guide users through basic configuration setup.

cu mulus@switch:~$ net example acl : access-list bgp : Border Gateway Protocol bond : Bond, port-channel, etc bridge : A layer2 bridge clag : Multi-Chassis Link Aggregation dot1x : Configure, Enable, Delete or Show IEEE 802.1X EAPOL link-settings : Physical link parameters lnv : Lightweight Network Virtualization management-vrf : Management VRF mlag : Multi-Chassis Link Aggregation ospf : Open Shortest Path First (OSPFv2) vlan-interfaces : IP interfaces for VLANs

cumulus@switch:~$ net example bridge

Scenario ======We are configuring switch1 and would like to configure the following - configure switch1 as an L2 switch for host-11 and host-12 - enable vlans 10-20 - place host-11 in vlan 10 - place host-12 in vlan 20 - create an SVI interface for vlan 10 - create an SVI interface for vlan 20 - assign IP 10.0.0.1/24 to the SVI for vlan 10 - assign IP 20.0.0.1/24 to the SVI for vlan 20 - configure swp3 as a trunk for vlans 10, 11, 12 and 20 swp3 *switch1 ------switch2 /\ swp1 / \ swp2 / \ / \ host-11 host-12

switch1 net commands ======

- enable vlans 10-20 switch1# net add vlan 10-20 - place host-11 in vlan 10 - place host-12 in vlan 20 switch1# net add int swp1 bridge access 10 switch1# net add int swp2 bridge access 20 - create an SVI interface for vlan 10 - create an SVI interface for vlan 20 - assign IP 10.0.0.1/24 to the SVI for vlan 10 - assign IP 20.0.0.1/24 to the SVI for vlan 20 switch1# net add vlan 10 ip address 10.0.0.1/24 switch1# net add vlan 20 ip address 20.0.0.1/24 - configure swp3 as a trunk for vlans 10, 11, 12 and 20 switch1# net add int swp3 bridge trunk vlans 10-12,20 # Review and commit changes switch1# net pending switch1# net commit

Verification ======switch1# net show interface switch1# net show bridge macs

Describe hardware abstraction Switchd

Switchd is the daemon at the heart of Cumulus Linux responsible for communicating between the switch and Cumulus Linux, and all the applications running on Cumulus Linux. The configuration for switchd is stored in /etc/cumulus/switchd.conf. Switchd peers directly with networking ASICs and normalizes the networking model, and peers with the kernel via netlink.

FIGURE 25

Netlink interaction with Kernel

Pros and cons of configuring user space vs. kernel

NCLU operates in the user space, and the user space versus kernel information is covered in the User Space and Kernel section. Netlink is a Linux kernel interface used for inter-process communication between the kernel and userspaces, as well as between different userspace processes.

Describe purpose of ONIE The Open Network Install Environment (ONIE) is an open source initiative that defines an open “install environment” for bare metal network switches. Open Network Install Environment (ONIE) is a small operating system, pre-installed as firmware on bare metal network switches that provides an environment for automated operating system provisioning.

ONIE enables a bare metal network switch ecosystem where switch hardware suppliers to manage their operations based on a small number of hardware SKUs, and end users have a choice among different network operating systems alternatives.

https://support.cumulusnetworks.com/hc/en-us/sections/200709257-ONIE

http://onie.org/

Describe how a system boots, how to install an OS https://opencomputeproject.github.io/onie/overview/index.html

When a new machine boots up for the first time, ONIE locates and executes a NOS vendor’s installation program.

FIGURE 26

ONIE is NOT used on every boot of the system. After the initial installation, subsequent boots go straight into the NOS, bypassing ONIE.

FIGURE 27

Mechanisms exist for a system to re-enter the installation phase. An API is defined so that network operating systems can direct the system to re-enter the installation phase.

ONIE search order and discovery

http://opencomputeproject.github.io/onie/design-spec/discovery.html

1. Statically configured (passed from boot loader)

2. Local file systems (USB for example)

3. Exact URLs from DHCPv4

4. Inexact URLs based on DHCP responses

5. IPv6 neighbors

6. TFTP waterfall

FIGURE 28

Cumulus Linux NOS installation options

https://docs.cumulusnetworks.com/display/DOCS/Installing+a+New+Cumulus+Linux+Image

1. DHCP/Web Server with DHCP Options

2. DHCP/Web Server without DHCP Options

3. Web Server without DHCP

4. FTP without a web server

5. Local File

6. USB Drive

Describe ZTP https://docs.cumulusnetworks.com/display/DOCS/Zero+Touch+Provisioning+-+ZTP

Zero touch provisioning (ZTP) enables you to deploy network devices quickly in large-scale environments. On first boot, Cumulus Linux invokes ZTP, which executes the provisioning automation used to deploy the device for its intended role in the network.

The provisioning framework allows for a one-time, user-provided script to be executed. You can develop this script using a variety of automation tools and scripting languages, providing flexibility to design the provisioning schemes to meet your needs.

While developing and testing the provisioning logic, you can use the ztp command in Cumulus Linux to manually invoke your provisioning script on a device.

Cumulus Linux ZTP options

1. Local File

2. USB Drive

3. DHCP

ZTP best practices (can expand each to tier 3 and apply details to each)

1. Install License

2. Test DNS Name Resolution

3. Check Cumulus Linux Release

4. Apply Management VRF Configuration

5. Perform Ansible Callback/Puppet or an agent install

6. Disable DHCP Hostname Override Setting

7. NCLU Usage

8. Test ZTP Scripts

Control plane protection ACL & SPAN with cl-acltool Netfilter ACLs, cl-acltool, iptables, & ebtables overview

Netfilter is the packet filtering framework in Cumulus Linux and most Linux distributions. Netfilter does not require a separate software daemon to run; it is part of the Linux kernel itself. Netfilter asserts policies at layers 2, 3, and 4 of the OSI model by inspecting packet and frame headers based on a list of rules. There are a number of tools available for configuring ACLs in Cumulus Linux with varying functions.

·· iptables, ip6tables, and ebtables are Linux userspace tools used to administer filtering rules for IPv4 packets, IPv6 packets, and Ethernet frames (layer 2 using MAC addresses)

·· NCLU is a Cumulus Linux-specific userspace tool used to configure custom ACLs

·· cl-acltool is a Cumulus Linux-specific userspace tool used to administer filtering rules and configure default ACLs

·· Without using cl-acltool, rules are NOT installed into hardware

·· Running cl-acltool -i (the installation option) resets all rules and deletes anything that is not stored in /etc/cumulus/acl/policy.conf

cumulus@switch:~$ sudo iptables -A INPUT -p icmp --icmp-type echo-request -j DROP

And the rules appear when you run cl-acltool -L:

cu mulus@switch:~$ sudo cl-acltool -L ip ------Listing rules of type iptables: ------TABLE filter : Chain INPUT (policy ACCEPT 72 packets, 5236 bytes) pkts bytes target prot opt in out source destination

0 0 DROP icmp -- any any anywhere anywhere icmp echo-request

However, running cl-acltool -i or reboot removes them. To ensure all rules that can be in hardware are hardware accelerated, place them in the /etc/cumulus/acl/policy.conf file, then runcl-acltool -i. Netfilter tables

When building rules to affect the flow of traffic, the individual chains can be accessed bytables . Linux provides three tables by default:

·· Filter classifies traffic or filters traffic (default table, if not specified)

·· Mangle alters packets as they move through the switch

·· NAT applies Network Address Translation rules (NOT supported in Cumulus Linux)

Each table has a set of default chains that can be used to modify or inspect packets at different points of the path through the switch. Chains contain the individual rules to influence traffic. Each table and the default chains they support are shown below. Tables and chains in green are supported by Cumulus Linux, and those in red are NOT supported (NO hardware acceleration) at this time.

FIGURE 29

Netfilter chains

Netfilter chains reference the position the action is taken during packet flow and processing. The rules created by these programs inspect or operate on packets at several points in the life of the packet through the system. These five points are known as chains:

·· PREROUTING touches packets before they are routed

·· INPUT touches packets after determination they are destined for the local system but before they are received by the control plane software

·· FORWARD touches transit traffic as it moves through the box

·· OUTPUT touches packets that are sourced by the control plane software before they are put on the wire

·· POSTROUTING touches packets immediately before they are put on the wire but after the routing decision has been made

FIGURE 30

Netfilter rules

Rules include a table, chain, match(es), a jump, and target(s).

FIGURE 31

·· Table: The first argument is thetable . The second example does not specify a table, because the filter table is implied if not specified.

·· Chain: The second argument is the chain. Each table supports several different chains.

·· Matches: The third argument(s) are called the matches. You can specify multiple matches in a single rule. However, the more matches you use in a rule, the more memory that rule consumes.

·· Jump: The jump specifies what action to take if the packet matches the rule. If jump is omitted in rule, then matching the rule will have no effect on the packet’s fate, but will increment counters.

·· Target(s): The target can be a user-defined chain, one of the special built-in targets that decides the fate of the packet immediately (like DROP), or an extended target. ACL rule assignment placement

·· If a switch port is assigned to a bond, any egress rules must be assigned to the bond.

·· When using the OUTPUT chain, rules must be assigned to the source. For example, if a rule is assigned to the switch port in the direction of traffic but the source is a bridge (VLAN), the traffic is not affected by the rule and must be applied to the bridge.

·· If all transit traffic needs to have a rule applied, use the FORWARD chain, not the OUTPUT chain. Describe control plane protection with cl-acltool

https://docs.cumulusnetworks.com/display/DOCS/Netfilter+-+ACLs#Netfilter-ACLs-ControlPlaneand DataPlaneTraffic

Control Plane traffic can be monitored and identified with tcdump. Cumulus Linux adds new extended targets to iptables and ebtables such as SPAN, ERSPAN, POLICE, TRICOLORPOLICE, and SETCLASS.

You can configure quality of service for traffic on both the control plane and the data plane. By using QoS policers, you can rate limit traffic so incoming packets get dropped if they exceed specified thresholds. Unfortunately, counters on POLICE ACL rules in iptables do not currently show the packets that are dropped due to those rules.

Using the POLICE target with iptables takes the following arguments:

·· --set-class sets the system internal class of service queue configuration to value.

·· --set-rate specifies the maximum rate in kilobytes (KB) or packets.

·· --set-burst specifies the number of packets or kilobytes (KB) allowed to arrive sequentially. Must be greater than or equal to 1.

·· --set-mode (KB|pkt) sets the mode in KB (kilobytes) or pkt (packets) for rate and burst size.

Cumulus comes with 2 files in/etc/cumulus/acl/policy.d/ for control plane filtering,00control_plane.rules and 99control_plane_catch_all.rules. These files can be edited or new files can be created numbered 01 through 98 for order processing. An excerpt of 00control_plane.rules is shown below.

cu mulus@leaf01:m g mt-vrf:~$ cat /etc/cumulus/acl/policy.d/00control _ plane.rules

INGRESS _ INTF = swp+

INGRESS _ CHAIN = INPUT

INNFWD _ CHAIN = INPUT,FORWARD

M A RTIA N _ SOURCES _ 4 = “240.0.0.0/5,127.0.0.0/8,224.0.0.0/4,255.255.255.255/32”

MARTIAN _ SOURCES _ 6 = “ff00::/8,::/128,::ffff:0.0.0.0/96,::1/128” CLAG _ PORT = 5342

BFD _ PORT = 3784

BFD _ ECHO _ PORT = 3785

BFD _ MH _ PORT = 4784

LNV _ CTRL _ PORT = 10001

MSDP _ PORT = 639

[ipta bles]

-A $INNFWD _ CHAIN --in-interface $INGRESS _ INTF -s $MARTIAN _ SOURCES _ 4 -j DROP

-A $INGRESS _ CHAIN --in-interface $INGRESS _ INTF -p udp --dport $BFD _ ECHO _ PORT -j SETCLASS --class 7 -A $INGRESS _ CHAIN -p udp --dport $BFD _ ECHO _ PORT -j POLICE --set-mode pkt --set- rate 2000 --set-burst 2000

-A $INGRESS _ CHAIN --in-interface $INGRESS _ INTF -p udp --dport $BFD _ PORT -j SETCLASS --class 7 -A $INGRESS _ CHAIN -p udp --dport $BFD _ PORT -j POLICE --set-mode pkt --set-rate 2000

--set-burst 2000

-A $INGRESS _ CHAIN --in-interface $INGRESS _ INTF -p udp --dport $BFD _ MH _ PORT -j SETCLASS --class 7 -A $INGRESS _ CHAIN -p udp --dport $BFD _ MH _ PORT -j POLICE --set-mode pkt --set-rate 2000 --set-burst 2000

-A $INGRESS _ CHAIN --in-interface $INGRESS _ INTF -p ospf -j SETCLASS --class 7 -A $INGRESS _ CHAIN -p ospf -j POLICE --set-mode pkt --set-rate 2000 --set-burst 2000

-A $INGRESS _ CHAIN --in-interface $INGRESS _ INTF -p pim -j SETCLASS --class 6 -A $INGRESS _ CHAIN -p pim -j POLICE --set-mode pkt --set-rate 2000 --set-burst 2000

-A $INGRESS _ CHAIN --in-interface $INGRESS _ INTF -p tcp --dport $MSDP _ PORT -j SETCLASS --class 6 -A $INGRESS _ CHAIN -p tcp --dport $MSDP _ PORT -j POLICE --set-mode pkt --set-rate 2000 --set-burst 2000 -A $INGRESS _ CHAIN --in-interface $INGRESS _ INTF -p tcp --sport $MSDP _ PORT -j SETCLASS --class 6 -A $INGRESS _ CHAIN -p tcp --sport $MSDP _ PORT -j POLICE --set-mode pkt --set-rate 2000 --set-burst 2000

Configure SPANs with cl-acltool SPAN overview

SPAN (Switched Port Analyzer) provides for the mirroring of all packets coming in from or going out of an interface (source) and being copied and transmitted out of a local port (destination) for monitoring. The SPAN destination port is also referred to as a mirror-to-port (MTP). The original packet is still switched, while a mirrored copy of the packet is sent out of the MTP.

ERSPAN (Encapsulated Remote SPAN) enables the mirrored packets to be sent to a monitoring node located anywhere across the routed network. The switch finds the outgoing port of the mirrored packets by doing a lookup of the destination IP address in its routing table. The original L2 packet is encapsulated with GRE for IP delivery.

SPAN and ERSPAN are configured via cl-acltool. The match criteria for SPAN and ERSPAN is usually an interface, but Selective SPAN can be used for granular match terms. The SPAN source interface can be a port, a subinterface or a bond interface. Ingress and egress (mellanox) traffic on interfaces can be matched.

Cumulus Linux supports a maximum of 2 SPAN destinations. The SPAN destination (MTP) interface can be a physical port, a subinterface, or a bond interface. The SPAN/ERSPAN action is independent of security ACL actions. If packets match both a security ACL rule and a SPAN rule, both actions will be carried out.

Limitations for SPAN | ERSPAN with Cumulus Linux

·· For Broadcom switches, Cumulus Linux supports a maximum of two SPAN destinations.

·· For Mellanox Spectrum switches, Cumulus Linux supports only a single SPAN destination in atomic mode or three SPAN destinations in non-atomic mode.

·· Multiple rules (SPAN sources) can point to the same SPAN destination, but a given SPAN source cannot specify two SPAN destinations.

·· To configure SPAN or ERSPAN on a Tomahawk switch, you must enable non-atomic update mode.

·· Mellanox switches reject SPAN ACL rules for an output interface that is a subinterface.

·· Mirrored traffic is not guaranteed. If the MTP is congested, mirrored packets might be discarded.

·· Cut-through mode is not supported for ERSPAN in Cumulus Linux on switches using Broadcom Tomahawk, Trident II+ and Trident II ASICs.

·· On Broadcom switches, SPAN does not capture egress traffic. Configuration steps

Create rules file in/etc/cumulus/acl/policy.d/

cu mulus@switch:~$ sudo bash -c ‘cat < /etc/cumulus/acl/policy.d/span.rules [ipta bles] -A FORWARD --in-interface swp4 -j SPAN --dport swp19 -A FORWARD --out-interface swp4 -j SPAN --dport swp19 EOF’

Verify installed rules

cu mulus@switch:~$ sudo iptables -L -v Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 DROP all -- swp+ any 240.0.0.0/5 anywhere 0 0 DROP all -- swp+ any loopback/8 anywhere 0 0 DROP all -- swp+ any base-address.mcast.net/8 anywhere 0 0 DROP all -- swp+ any 255.255.255.255 anywhere 0 0 SETCLASS ospf -- swp+ any anywhere anywhere SETCLASS class:7 0 0 POLICE ospf -- any any anywhere anywhere POLICE mode:pkt rate:2000 burst:2000 0 0 SETCLASS tcp -- swp+ any anywhere anywhere tcp dpt:bgp SETCLASS class:7 0 0 POLICE tcp -- any any anywhere anywhere tcp dpt:bgp POLICE mode:pkt rate:2000 burst:2000 0 0 SETCLASS tcp -- swp+ any anywhere anywhere tcp spt:bgp SETCLASS class:7

0 0 POLICE tcp -- any any anywhere anywhere tcp spt:bgp POLICE mode:pkt rate:2000 burst:2000 0 0 SETCLASS tcp -- swp+ any anywhere anywhere tcp dpt:5342 SETCLASS class:7 0 0 POLICE tcp -- any any anywhere anywhere tcp dpt:5342 POLICE mode:pkt rate:2000 burst:2000 0 0 SETCLASS tcp -- swp+ any anywhere anywhere tcp spt:5342 SETCLASS class:7 0 0 POLICE tcp -- any any anywhere anywhere tcp spt:5342 POLICE mode:pkt rate:2000 burst:2000 0 0 SETCLASS icmp -- swp+ any anywhere anywhere SETCLASS class:2 0 0 POLICE icmp -- any any anywhere anywhere POLICE mode:pkt rate:100 burst:40 15 5205 SETCLASS udp -- swp+ any anywhere anywhere udp dpts:bootps:bootpc SETCLASS class:2 11 3865 POLICE udp -- any any anywhere anywhere udp dpt:bootps POLICE mode:pkt rate:100 burst:100 0 0 POLICE udp -- any any anywhere anywhere udp dpt:bootpc POLICE mode:pkt rate:100 burst:100 0 0 SETCLASS tcp -- swp+ any anywhere anywhere tcp dpts:bootps:bootpc SETCLASS class:2 0 0 POLICE tcp -- any any anywhere anywhere tcp dpt:bootps POLICE mode:pkt rate:100 burst:100 0 0 POLICE tcp -- any any anywhere anywhere tcp dpt:bootpc POLICE mode:pkt rate:100 burst:100 17 1088 SETCLASS igmp -- swp+ any anywhere anywhere SETCLASS class:6 17 1156 POLICE igmp -- any any anywhere anywhere POLICE mode:pkt rate:300 burst:100 394 41060 POLICE all -- swp+ any anywhere anywhere ADDRTYPE match dst-type LOCAL POLICE mode:pkt rate:1000 burst:1000 class:0 0 0 POLICE all -- swp+ any anywhere anywhere ADDRTYPE match dst-type IPROUTER POLICE mode:pkt rate:400 burst:100 class:0 988 279K SETCLASS all -- swp+ any anywhere anywhere SETCLASS class:0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 DROP all -- swp+ any 240.0.0.0/5 anywhere 0 0 DROP all -- swp+ any loopback/8 anywhere 0 0 DROP all -- swp+ any base-address.mcast.net/8 anywhere 0 0 DROP all -- swp+ any 255.255.255.255 anywhere

26864 4672K SPAN all -- swp4 any anywhere anywhere dport:swp19 <---- input packets on swp4

40722 47M SPAN all -- any swp4 anywhere anywhere dport:swp19 <---- output packets on swp4

Chain OUTPUT (policy ACCEPT 67398 packets, 5757K bytes) pkts bytes target prot opt in out source destination

Install rules

cu mulus@switch:~$ sudo cl-acltool -i [sudo] password for cumulus: Reading rule file /etc/cumulus/acl/policy.d/00control _ plane.rules ... Processing rules in file /etc/cumulus/acl/policy.d/00control _ plane.rules ... Reading rule file /etc/cumulus/acl/policy.d/99control _ plane _ catch _ all.rules ... Processing rules in file /etc/cumulus/acl/policy.d/99control _ plane _ catch _ all.rules ... Reading rule file /etc/cumulus/acl/policy.d/span.rules ... Processing rules in file /etc/cumulus/acl/policy.d/span.rules ... Installing acl policy done.

Verify SPAN rules were installed properly

cu mulus@switch:~$ sudo cl-acltool -L all | grep SPAN 38025 7034K SPAN all -- swp4 any anywhere anywhere dport:sw p19 50832 55M SPAN all -- any swp4 anywhere anywhere dport:sw p19

Rule removal

# Remove rules file: cu mulus@switch:~$ sudo rm /etc/cumulus/acl/policy.d/span.rules

# Reload the default rules cu mulus@switch:~$ sudo cl-acltool -i

# To verify that the SPAN rules were removed: cu mulus@switch:~$ sudo cl-acltool -L all | grep SPAN

Selective SPAN

https://docs.cumulusnetworks.com/display/DOCS/Network+Troubleshooting#NetworkTroubleshooting- selective_spanning

SPAN/ERSPAN traffic rules can be configured to limit the traffic that is spanned to reduce the volume of copied data. Cumulus Linux supports selective spanning for iptables only. ip6tables and ebtables are NOT supported. The following matching fields are supported:

·· IPv4 SrcIP/DstIP

·· IP protocol

·· L4 (TCP/UDP) src/dst port

·· TCP flags

·· An ingress port/wildcard (swp+) can be specified in addition

With ERSPAN, a maximum of two --src-ip --dst-ip pairs are supported. Exceeding this limit produces an error when you install the rules with cl-acltool.

Layer 2 ebtables ACL example

The following rule blocks any traffic with source MAC address 00:00:00:00:00:12 and destination MAC address 08:9e:01: ce: e2:04 going from any switch port egress/ingress.

[ebtables] -A FORWARD -s 00:00:00:00:00:12 -d 08:9e:01:ce:e2:04 -j DROP

SPAN with Tomahawk based switches

In Cumulus Linux, atomic update mode is enabled by default. If you have Tomahawk switches and plan to use SPAN and/or mangle rules, you must disable atomic update mode in the file/etc/cumulus/switchd.conf , then restart switchd.

acl.non _ atomic _ update _ mode = TRUE

cumulus@switch:~$ sudo systemctl restart switchd.service

Other references

Hardware Limit — https://docs.cumulusnetworks.com/display/DOCS/Netfilter+-+ACLs#Netfilter-ACLs-Hard wareLimitationsonNumberofRules

Support Rule Types — https://docs.cumulusnetworks.com/display/DOCS/Netfilter+-+ACLs#Netfilter-ACLs- supportedSupportedRuleTypes

IPv6 and services IPv6

An overview of IPv6 is covered in the IPv6 Overview section earlier in the document.

AAA

Cumulus Networks offers add-on packages that enable RADIUS users to log in to Cumulus Linux switches in a transparent way with minimal configuration. There is no need to create accounts or directories on the switch. Authentication is handled with PAM and includes login, ssh, sudo and su. The general steps needed to enable RADIUS are:

·· Install RADIUS Packages

·· Configure RADIUS Client

·· Enable Login Without Local Accounts

·· Enable Local Fallback Authentication

·· Verify RADIUS Client Configuration RADIUS package install

The RADIUS packages are not included in the base Cumulus Linux image; there is no RADIUS metapackage.

cumulus@switch:~$ sudo apt-get update cumulus@switch:~$ sudo apt-get install libnss-mapuser libpam-radius-auth

The libpam-radius-auth package supplied with the Cumulus Linux RADIUS client is a newer version than the one in Debian Jessie. This package has added support for IPv6, the src_ip option described below, as well as a number of bug fixes and minor features. The package also includes VRF support, provides man pages describing the PAM and RADIUS configuration, and sets the SUDO_PROMPT environment variable to the login name for RADIUS mapping support.

The libnss_mapuser package is specific to Cumulus Linux and supports the getgrent, getgrnam and getgrgid library interfaces. These interfaces add logged in RADIUS users to the group member list for groups that contain the mapped_user (radius_user) if the RADIUS account is unprivileged and add privileged RADIUS users to the group member list for groups that contain the mapped_priv_user (radius_priv_user) during the group lookups.

·· The PAM configuration is modified automatically using pam-auth-update (8), and the NSS configuration file/etc/nsswitch.conf is modified to add the mapuser and mapuid plugins. If you remove or purge the packages, these files are modified to remove the configuration for these plugins.

·· The radius_shell package is added, which installs the /sbin/radius_shell and setcap cap_setuid program used as the login shell for RADIUS accounts. The package adjusts the UID when needed, then runs the bash shell with the same arguments. When installed, the package changes the shell of the RADIUS accounts to /sbin//radius_shell, and to /bin/shell if the package is removed. This package is required for privileged RADIUS users to be enabled. It is not required for regular RADIUS client use.

·· The radius_user account is added to the netshow group and the radius_priv_user account to the netedit and sudo groups. This change enables all RADUS logins to run NCLU net show commands and all privileged RADIUS users to also run net add, net del, and net commit commands, and to use sudo.

After installation is complete, either reboot the switch or run the sudo systemctl restart netd command.

RADIUS client configuration with local fallback

To configure the RADIUS client, edit the/etc/pam_radius_auth.conf file:

·· Add the hostname or IP address of at least one RADIUS server (such as a freeradius server on Linux) and the shared secret used to authenticate and encrypt communication with each server.

·· Multiple server configuration lines are verified in the order listed. Other than memory, there is no limit to the number of RADIUS servers that can be used.

·· The server port number or name is optional. The system looks up the port in the /etc/services file. However, you can override the ports in the /etc/pam_radius_auth.conf file.

·· If the server is slow or latencies are high, change the timeout setting. The setting defaults to 3 seconds.

·· If you want to use a specific interface to reach the RADIUS server, specify the src_ip option. You can specify the hostname of the interface, an IPv4, or an IPv6 address. If you specify the src_ip option, you must also specify the timeout option.

·· Set the vrf-name field. This is typically set to mgmt if you are using a management VRF. You cannot specify more than one VRF.

The configuration file includes the mapped_priv_user field that sets the account used for privileged RADIUS users and the priv-lvl field that sets the minimum value for the privilege level to be considered a privileged login (the default value is 15). If you edit these fields, make sure the values match those set in the /etc/nss_mapuser.conf file.

The following example provides a sample /etc/pam_radius_auth.conf file configuration:

mapped _ priv _ user radius _ priv _ user # server[:port] shared _ secret timeout (secs) src _ ip 192.168.0.254 secretkey other-server othersecret 3 192.168.1.10 # when mgmt vrf is in use vrf-name mgmt

Debugging messages are written to /var/log/syslog. When the RADIUS client is working correctly, comment out the debug line.

To configure local authentication fallback:

·· Add a local privileged user account with same unique identifier as the privileged radius user

·· Enable local privileged user to run appropriate commands

·· Edit the /etc/passwd file to move the local user line before the radius entry

·· Set the local password for the user

The example shows the setting based on the radius_priv_user account in the /etc/passwd file is radius_priv_user:x:1002:1001::/home/radius_priv_user:/sbin/radius_shell.

cumulus@switch:~$ sudo useradd -u 1002 -g 1001 -o -s /sbin/radius _ shell craigtompkins

cumulus@switch:~$ sudo adduser craigtompkins netedit cumulus@switch:~$ sudo adduser craigtompkins sudo cumulus@switch:~$ sudo systemctl restart netd

cumulus@switch:~$ sudo vi /etc/passwd ... craigtompkins:x:1002:1001::/home/craigtompkins:/sbin/radius _ shell radius _ priv _ user:x:1002:1001::/home/radius _ priv _ user:/sbin/radius _ shell

cumulus@switch:~$ sudo passwd craigtompkins

Enabling login without local accounts

Cumulus Linux handles this transparently once the lbnss-mapuser package is installed. Specific details can be found here. https://docs.cumulusnetworks.com/display/DOCS/RADIUS+AAA#RADIUSAAA- EnableLoginwithoutLocalAccounts

RADIUS client configuration verification

To verify that the RADIUS client is configured correctly, log in as a non-privileged user and run a net add interface command.

In this example, the ops user is not a priveleged RADIUS user so they cannot add an interface, while the admin user is a privileged RADIUS user (with privilege level 15) so is able to add interface swp1.

ops@leaf01:~$ net add interface swp1 ERROR: User ops does not have permission to make networking changes.

admin@leaf01:~$ net add interface swp1 admin@leaf01:~$ net pending --- /etc/network/interfaces 2018-04-06 14:49:33.099331830 +0000

+++ /var/run/nclu/iface/interfaces.tmp 2018-04-06 16:01:16.057639999 +0000 @@ -3,10 +3,13 @@

source /etc/network/interfaces.d/*.intf

# The loopback network interface auto lo iface lo inet loopback

# The primary network interface auto eth0 iface eth0 inet dhcp + +auto swp1 +iface swp1 ...

Network Time Protocol (NTP)

https://docs.cumulusnetworks.com/display/DOCS/Setting+Date+and+Time

NTP synchronizes time between devices configured in a client-server relationship. Cumulus will use eth0 by default, but this can be changed.

Prior to NTP configuration validate the time zone acceptable( values) and datetime and change them if they need modification. The /etc/timezone file can be modified or replaced. Running the second command will make the changes take effect immediately.

cumulus@switch:~$ cat /etc/timezone US/Eastern

cumulus@switch:~$ sudo dpkg-reconfigure --frontend noninteractive tzdata

To set the system clock according to the time zone configured:man date(1)

cumulus@switch$ sudo date -s “Tue Jan 12 00:37:13 2016”

To write the current value of the system (software) clock to the hardware clock: man hwclock(8)

cumulus@switch$ sudo hwclock -w

NTP with DHCP

If you use DHCP and want to specify your NTP servers, you must specify an alternate configuration file for NTP.

Before you create the file, ensure that the DHCP-generated configuration file exists. In Cumulus Linux 3.6.1 and later (which uses NTP 1:4.2.8), the DHCP-generated file is named/run/ntp.conf . This file is generated by the /etc/dhcp/dhclient-exit-hooks.d/ntp script and is a copy of the default /etc/ntp.conf with a modified server list from the DHCP server. If this file does not exist and you plan on using DHCP in the future, you can copy your current /etc/ntp.conf file to the location of the DHCP file.

To use an alternate configuration file that persists across upgrades of Cumulus Linux, create a systemd unit override file called/etc/systemd/system/ntp.service.d/config.conf and add the following content:

cumulus@switch:~$ sudo echo ‘ [Ser vice] ExecStart= ExecStart=/usr/sbin/ntpd -n -u ntp:ntp -g -c /run/ntp.conf.dhcp ‘ > ~/over sudo mkdir -p /etc/systemd/system/ntp.service.d sudo mv ~/over /etc/systemd/system/ntp.service.d/dhcp.conf sudo chown root:root /etc/systemd/system/ntp.service.d/dhcp.conf

Reload services and check ntp.service status.

cumulus@switch:~$ sudo systemctl daemon-reload cumulus@switch:~$ sudo systemctl restart ntp cumulus@switch:~$ sudo systemctl status -n0 ntp.service

With this unit file override present, changing NTP settings using NCLU doNOT take effect until the DHCP script regenerates the alternate NTP configuration file.

NTP via NCLU

The ntpd daemon running on the switch implements the NTP protocol. It synchronizes the system time with time servers listed in /etc/ntp.conf. The ntpd daemon is started at boot by default. You can specify the NTP server or servers you want to use with NCLU; include the iburst option to increase the sync speed.

cumulus@switch:~$ net add time ntp server 4.cumulusnetworks.pool.ntp.org iburst cumulus@switch:~$ net pending cumulus@switch:~$ net commit

These commands add the NTP server 4.cumulusnetworks.pool.ntp.org to the list of servers in /etc/ntp.conf. Servers 0 through 3 are included by default.

# pool.ntp.org maps to about 1000 low-stratum NTP servers. Your server will # pick a different set every time it starts up. Please consider joining the # pool: server 0.cumulusnetworks.pool.ntp.org iburst server 1.cumulusnetworks.pool.ntp.org iburst server 2.cumulusnetworks.pool.ntp.org iburst server 3.cumulusnetworks.pool.ntp.org iburst server 4.cumulusnetworks.pool.ntp.org iburst

SNMP historical overview & components

https://docs.cumulusnetworks.com/display/DOCS/Simple+Network+Management+Protocol+ %28SNMP%29+Monitoring

SNMP is an IETF RFC standards-based network management architecture and protocol tracing back to Carnegie-Mellon University. Subsequently modified by programmers at the University of California, the code was made publicly available under the name ucd-snmp. This was further extended by the University of Liverpool as well as in Denmark. The name update to net-snmp and became a fully-fledged collaborative open source project. The version used by Cumulus Networks is the latest net-snmp 5.7 branch with added custom MIBs, and pass-through and pass-persist scripts.

SNMP Management servers gather information from different systems in a consistent manner and its longevity is due to standardizing the objects collected from devices, the protocol used for transport, and architecture of the management systems. The most widely used versions of SNMP are v1 and v2c, while v3 uses advanced security features and is the recommended choice.

Agents — The SNMP agents (snmpd) on the switches perform the bulk of the work by gathering information about the local system and storing it in an internal database called the management information base (MIB). The MIB is a standardized, hierarchical structure storing information to be queried. Parts of the MIB tree are available and provided to incoming requests originating from an NMS host that has authenticated with the correct credentials. You can configure the Cumulus Linux switch with usernames and credentials to provide authenticated and encrypted responses to NMS requests. The snmpd agent can also proxy requests and act as a master agent to sub-agents running on other daemons (FRR, LLDP).

Managers — An SNMP Network Management System (NMS) is a computer that is configured to poll SNMP agents to gather and present information. The manager can be any machine capable of sending query requests to SNMP agents with the correct credentials. The NMS can be a large monitoring suite or a simple set of scripts that collect through polling the agents and present the data. SNMP agents can also send unsolicited Traps/Inform messages to the SNMP Manager based on predefined criteria.

Management Information Base (MIB) — The MIB is a database implemented on the daemon or agent and follows IETF RFC standards the same as the manager. It is a hierarchical structure that, in many areas, is globally standardized, but also flexible enough to allow vendor-specific additions. Cumulus Networks implements a number of custom enterprise MIB tables and these are defined in text files located on the switch and in files named/usr/share/snmp/mibs/Cumulus* . The MIB structure is best understood as a top- down hierarchical tree where each branch that forks is labeled both with an identifying number (starting at 1) and an identifying string unique to that level of the hierarchy. These strings and numbers can be used interchangeably in order for a specific node of the tree to be traced from the unnamed root of the tree to the node in question.

Object Identifier (OID) — The parent IDs (numbers or strings) are strung together, starting with the most general to form an address for the MIB Object with each junction in the hierarchy represented by a dot. The series of ID strings or numbers separated by dots is an address known as an object identifier (OID).

Cumulus custom MIBs

No changes are required in the /etc/snmp/snmpd.conf file on the switch to support the custom Cumulus Networks MIBs. The following lines are already included by default and provide support for both the Cumulus Counters and the Cumulus Resource Query MIBs.

sysObjectID 1.3.6.1.4.1.40310 pass _ persist .1.3.6.1.4.1.40310.1 /usr/share/sn m p/resq _ pp.py pass _ persist .1.3.6.1.4.1.40310.2 /usr/share/sn m p/cl _ d rop _ cntrs _ pp.py

However, you need to copy several files to the NMS server(s) for the custom Cumulus MIB to be recognized on the NMS server.

/usr/share/snmp/mibs/Cumulus-Snmp-MIB.txt /usr/share/snmp/mibs/Cumulus-Counters-MIB.txt /usr/share/snmp/mibs/Cumulus-Resource-Query-MIB.txt

SNMP configuration with NCLU

Cumulus Linux 3.4 and later releases support configuring SNMP with NCLU. While NCLU does not have 100% coverage to configure every single snmpd feature, it is therecommended method of configuring snmpd. You are not restricted to using NCLU for configuration and can edit the/etc/snmp/snmpd.conf file and control snmpd with systemctl commands. For Cumulus Linux versions earlier than 3.0, snmpd has a default configuration that listens to incoming requests on all interfaces.

·· Remove previous configuration (if warranted)

·· Configure SNMP Agent (snmpd)

·· Assign IP address to interface

·· Assign IP address to SNMP listener (default 127.0.0.1)

·· For SNMPv3, username, password, and encryption settings

·· If using SNMPv1 or SNMPv2c, create default community string

Many options are available for review and implementation, below is an example of SNMPv3 tied to the mgmt VRF using MD5 and reporting link-up, link-down, and authentication failure traps.

net del snmp-server all net add snmp-server listening-address all vrf mgmt

net add snmp-server username craigtompkins auth-md5 craigspassword encrypt-aes aessecret

net add snmp-server trap-destination 20.20.20.20 vrf mgmt username craigtompkins auth-md5 craigspassword encrypt-aes aessecret engine-id 0x80001f888070939b14a514da5a00000000 inform

net add snmp-server trap-link-up check-frequency 15 net add snmp-server trap-link-down check-frequency 10 net add snmp-server trap-snmp-auth-failures

net add snmp-server trap-cpu-load-average one-minute 45.0 five-minute 30.00 fifteen- minute 20.00

Inform keywords in the trap definition use SNMPv3 acknowledged informs rather than traps. NCLU restarts the snmpd daemon after configuration changes are made and committed.

To use file manipulation and replacement for manual or automated SNMP configuration, see https://docs. cumulusnetworks.com/display/DOCS/Simple+Network+Management+Protocol+%28SNMP%29+Monitoring #SimpleNetworkManagementProtocol(SNMP)Monitoring-ConfigureSNMPManually.

DHCP relay overview

DHCP Relay provides a forwarding action for DHCP broadcast packets, which would otherwise be dropped at a layer 2/3 demarcation point, towards a set of layer 3 unicast addresses. This allows for centralized DHCP servers to serve an entire site or organization.

DHCP and DHCP Relay daemons and disabled by default. After configuring the services needed, the appropriate service(s) will need to be restarted. Both IPv4 and IPv6 DHCP Relays are supported but must be started separately.

DHCP relay IPV4 configuration with NCLU

The following example shows the use of the command to set IP address 10.0.0.1 on the loopback interface as the giaddr and creates the latter configuration in the/etc/default/isc-dhcp-relay file:

cu mulus@leaf01:~$ net add dhcp relay giaddr-interface lo 10.0.0.1

cu mulus@leaf01:~$ cat /etc/default/isc-dhcp-relay ... # Additional options that are passed to the DHCP relay daemon? OPTIONS=”-U 10.0.0.1%lo”

When using VRR for redundancy, the configuration procedure for DHCP relay is the same, except that DHCP relay must run on the SVI and not on the -v0 interface.

DHCP relay IPv6 configuration

NCLU does not support IPv6 DHCP Relay configuration as of version 3.7.3.

Edit the /etc/default/isc-dhcp-relay6 file has a different format than the/etc/default/isc-dhcp-relay file for IPv4 DHCP relays. Make sure to configure the variables appropriately by editing this file.

cu mulus@leaf01:$ sudo nano /etc/default/isc-dhcp-relay6 SERVERS=” -u 2001:db8:100::2%swp51 -u 2001:db8:100::2%swp52” INTF _ CMD=”-l vlan1”

After you finish configuring the DHCP relay, save your changes, restart the dhcrelay6 service, then enable the dhcrelay6 service so the configuration persists between reboots:

cumulus@leaf01:~$ sudo systemctl restart dhcrelay6.service cumulus@leaf01:~$ sudo systemctl enable dhcrelay6.service

To see the status of the IPv6 DHCP relay, use the systemctl status dhcrelay6.service command:

cu mulus@leaf01:~$ sudo systemctl status dhcrelay6.service ● dhcrelay6.service - DHCPv6 Relay Agent Daemon Loaded: loaded (/lib/systemd/system/dhcrelay6.service; disabled) Active: active (running) since Fri 2016-12-02 21:00:26 UTC; 1s ago Docs: man:dhcrelay(8) Main PID: 6152 (dhcrelay) CGroup: /system.slice/dhcrelay6.service └─6152 /usr/sbin/dhcrelay -6 --nl -d -q -l vlan1 -u 2001:db8:100::2 swp51 -u 2001:db8:100::2 swp52

DHCP relay troubleshooting

Use the journalctl command to look at the behavior on the Cumulus Linux switch that is providing the DHCP relay functionality:

cu mulus@leaf01:~$ sudo journalctl -l -n 20 | grep dhcrelay Dec 05 20:58:55 leaf01 dhcrelay[6152]: sending upstream swp52 Dec 05 20:58:55 leaf01 dhcrelay[6152]: sending upstream swp51 Dec 05 20:58:55 leaf01 dhcrelay[6152]: Relaying Reply to fe80::4638:39ff:fe00:3 port 546 down. Dec 05 20:58:55 leaf01 dhcrelay[6152]: Relaying Reply to fe80::4638:39ff:fe00:3 port 546 down. Dec 05 21:03:55 leaf01 dhcrelay[6152]: Relaying Renew from fe80::4638:39ff:fe00:3 port 546 going up. Dec 05 21:03:55 leaf01 dhcrelay[6152]: sending upstream swp52 Dec 05 21:03:55 leaf01 dhcrelay[6152]: sending upstream swp51 Dec 05 21:03:55 leaf01 dhcrelay[6152]: Relaying Reply to fe80::4638:39ff:fe00:3 port 546 down. Dec 05 21:03:55 leaf01 dhcrelay[6152]: Relaying Reply to fe80::4638:39ff:fe00:3 port 546 down.

Journalctl command with the --since flag specifies a time period:

cu mulus@leaf01:~$ sudo journalctl -l --since “2 minutes ago” | grep dhcrelay Dec 05 21:08:55 leaf01 dhcrelay[6152]: Relaying Renew from fe80::4638:39ff:fe00:3 port 546 going up. Dec 05 21:08:55 leaf01 dhcrelay[6152]: sending upstream swp52 Dec 05 21:08:55 leaf01 dhcrelay[6152]: sending upstream swp51

If experiencing issues with the DHCP relay, run the following commands to determine if the issue is with systemd. The following commands manually activate the DHCP relay process and they do not persist when you reboot the switch:

cu mulus@switch:~$ /usr/sbin/dhcrelay -4 -i -i cu mulus@switch:~$ /usr/sbin/dhcrelay -6 -l -u %

cu mulus@leaf01:~$ /usr/sbin/dhcrelay -4 -i vlan1 172.16.1.102 -i swp51 cu mulus@leaf01:~$ /usr/sbin/dhcrelay -6 -l vlan1 -u 2001:db8:100::2%swp51

DHCP relay RFC 3527 link selection sub-option

A unique IP is required from the device relaying the DHCP request representing the gateway IP address (giaddr) to identify the requesting subnet. This poses an issue when Anycast is used with the same gateway across multiple devices as with EVPN. Cumulus Linux supports RFC 3527 which allows the specification of link selection sub-option to identify the subnet when the giaddr is insufficient.

FIGURE 32

Design concepts

Describe clos design Clos is an architecture developed between the 1930 and 1950s, made popular by Charles Clos to meet scalability requirements in circuit switched networks. With the explosion of scale in modern data center networks, the Clos design is being utilized for similar purposes.

For data center networks the clos design is represented by leaf and spine devices, where servers connect to leafs, each leaf connects to each spine, spines do not connect to each other, and leafs only connect to each other in support of device virtualization techniques such as MLAG. This provides consistency in bandwidth and latency between servers as the majority of traffic in the data center is now east and west, rather than north and south.

Describe various modern architecture designs https://docs.cumulusnetworks.com/display/DOCS/Data+Center+Host+to+ToR+Architecture

https://cumulusnetworks.com/media/resources/validated-design-guides/Cumulus-Linux-Layer-2-HA- Validated-Design-Guide_v1.0.0.pdf

https://cumulusnetworks.com/media/cumulus/pdf/technical/validated-design-guides/Big-Data-Cumulus- Linux-Validated-Design-Guide.pdf

Traditional spanning tree — single attached

Single attached means that each server is connected to a single switch.

Positively, this is an established topology, widely supported with multiple vendors, good documentation and easy configuration. This topology will provide Layer 2 reachability and allow the use of spanning tree commands.

Negatively, this topology will not provide failover capability, waste half of available bandwidth, and the switches can see multiple MAC addresses.

MLAG

FIGURE 33

MLAG reduces the dependence on spanning tree and enables 100% bandwidth utilization, but is vendor specific without interoperability, requires an inter-switch link (ISL), more configuration, and can be more complex.

FIGURE 34

L3 single-attached hosts

FIGURE 35

The server (physical host) has one or more links to only one ToR switch.

Layer 3 architecture with single-attached hosts provides simple networking configuration with no spanning tree, MLAG, no L2 loops, no leaf inter-switch links. All of which allow for great route scaling and flexibility.

Unfortunately, this simplistically comes at the cost of redundancy for server connectivity and uses a single top of rack switch as its gateway. It is the responsibility of the remaining infrastructure to overcome the loss of the rack.

Redistributed neighbor vs. Routing on Host (ROH)

With redistribute neighbor, a daemon grabs ARP entries dynamically, and utilizes redistribute table for FRRouting to enter them into the fabric. This solution is limited to IPv4, provides no layer 2 adjacency without VXLAN, host traffic dependent on proxy ARP, and withdrawal from original leaf could be up to 4 hours. First hop redundancy is handled with equal cost routes on the host to both top of rack switches. This solution allows containers to be built and destroyed with their Layer 3 information dynamically introduced to the network.

FIGURE 36

Routing on the host means there is a routing application (such as FRRouting) either on the bare metal host (no VMs/containers) or the hypervisor (Ubuntu with KVM). This is preferred and recommended by Cumulus Networks over redistribute neighbor.

There is no requirement for MLAG, no spanning tree or layer 2 domain, no loops, and 3 or more top of rack devices can be used. Host and VM mobility is enabled and traffic engineering can be used to allow for hardware and software upgrades. This solution is dependent on host routing support, and would require VXLAN for layer 2 adjacencies.

FIGURE 37

Routing on the Virtual Machine (VM)

Routing on the VM is very similar to routing on the host and includes those benefits, but instead of routing on the hypervisor, each virtual machine utilizes its own routing stack. This removes the need for the hypervisor platform to support routing, but all VMs must support routing, and 1 router per VM can create scaling issues.

FIGURE 38

Virtual router

Virtual router (vRouter) runs as a VM on the hypervisor/host, sends routes to the ToR using BGP or OSPF. In addition to routing on host; virtual router enables same rack Multi-tenancy and the base OS does not need to be routing capable. The virtual router acts as a local gateway and provides multiple routes through top of rack switches. Older Linux kernels will not provide per flow ECMP.

https://support.cumulusnetworks.com/hc/en-us/articles/213177027-Installing-the-Cumulus-Linux-Quagga- Package-on-an-Ubuntu-Server

FIGURE 39

Routing on the Host (ROTH) BGP advertisement best practices

Limiting the exchange of routing information at various parts in the network is a best practice to follow. One way to do this is show in the image.

FIGURE 40

Anycast with manual redistribution

In contrast to routing on the host (preferred), this method allows a user to route to the host. The ToRs are the gateway, as with redistribute neighbor, except because there is no daemon running. The networks must be manually configured under the routing process. There is potential ease to black hole traffic unless a script is run to remove the routes when the host no longer responds.

FIGURE 41

This provides most benefits of routing on the host without host routing or redistribute neighbor requirement. Moving subnets from one top of rack to another is manual process and requires synchronization between server and network teams.

LNV with MLAG

https://docs.cumulusnetworks.com/display/DOCS/Lightweight+Network+Virtualization+Overview

FIGURE 42

Lightweight Network Virtualization (LNV) is a technique for deploying VXLANs without a central controller on bare metal switches. This solution requires no external controller or software suite; it runs the VXLAN service and registration daemons on Cumulus Linux itself. The data path between bridge entities is established on top of a layer 3 fabric by means of a simple service node coupled with traditional MAC address learning.

The host runs LACP (Etherchannel/bond) to the pair of ToRs. LNV (Lightweight Network Virtualization) then transports the Layer 2 bridges across a Layer 3 fabric. The layer 2 domain is limited to the local top of rack switches, and the aggregation is all layer 3 providing great route scaling and flexibility, and high availability.

LNV is exclusive of EVPN and they CANNOT be used together.

Centralized VXLAN routing with bridging using EVPN design reference

FIGURE 43

Distributed VXLAN routing with asymmetric model design reference

FIGURE 44

Distributed VXLAN routing with symmetric model design reference

FIGURE 45

Describe service leafs Dedicated switch device(s) that peer the Clos network to an outside network, also referred to as a border, edge, or exit leaf. These devices isolate the inside from the outside of the data center network and should remove any private ASNs or information and aggregate the internal routes towards the external networks. The exit or service leafs provide services outside.

Describe Equal Cost Multipath (ECMP) https://docs.cumulusnetworks.com/display/DOCS/Equal+Cost+Multipath+Load+Sharing+- +Hardware+ECMP

ECMP overview

Cumulus Linux supports hardware-based equal cost multipath (ECMP) load sharing. ECMP is enabled by default and load sharing occurs automatically for all routes with multiple next hops installed. ECMP load sharing supports both IPv4 and IPv6 routes.

For routes to be considered equal they must:

·· Originate from the same routing protocol. Routes from different sources are not considered equal. For example, a static route and an OSPF route are not considered for ECMP load sharing.

·· Have equal cost. If two routes from the same protocol are unequal, only the best route is installed in the routing table.

ECMP hashing

ECMP hashing does not keep a record of packets that have hashed to each next hop and does not guarantee that traffic sent to each next hop is equal.

Cumulus Linux hashes on the following fields:

·· IP protocol

·· Ingress interface

·· Source IPv4 or IPv6 address

·· Destination IPv4 or IPv6 address

·· Source port (TCP|UDP)

·· Destination port (TCP|UDP)

FIGURE 46

ECMP resilient hashing

In Cumulus Linux, when a next hop fails or is removed from an ECMP pool, the hashing or hash bucket assignment can change. For deployments where there is a need for flows to always use the same next hop, like TCP anycast deployments, this can create session failures. Resilient hashing helps prevent disruptions when next hops are added but does not assist when next hops are added. When one next hop fails, other next hops are filled in its place to maintain the fixed total number of buckets.

The ECMP hash performed with resilient hashing is exactly the same as the default hashing mode, and only the method in which next hops are assigned to hash buckets differs. Resilient Hashing is disabled by default but does support both IPv4 and IPv6 routes.

Resilient Hashing is supported on switches with Broadcom Tomahawk, Trident II, Trident II+, Trident 3, and Mellanox Spectrum chipsets.

When resilient hashing is enabled, 65,536 buckets are created to be shared among all ECMP groups, and next hops are assigned in a round robin fashion. An ECMP group is a list of unique next hops that are referenced by multiple ECMP routes. The number of buckets can be configured as 64, 128, 256, 512 or 1024; the default is 128:

Number of Hash Buckets Number of Supported ECMP Groups

64 1024

128 512

256 256

512 128

1024 64

ECMP resilient hashing configuration

To enable resilient hashing, the file/etc/cumulus/datapath/traffic.conf must be edited. You may optionally modify number of hash buckets, which affects the supported number of ECMP groups per the chart. Then restart switchd.service.

# Enable resilient hashing resilient _ hash _ enable = TRUE

# Resilient hashing flowset entries per ECMP group # Valid values - 64, 128, 256, 512, 1024 resilient _ hash _ entries _ ecmp = 256

cumulus@switch:~$ sudo systemctl restart switchd.service

Describe oversubscription ratios Describe port density, sizing of the DC

Spine and Leaf Clos networks are often built with a 4 to 1 uplink to downlink speed ratio in the leafs in direct port comparison, but with 48 downlink ports and 4, 6, and 8 uplink ports are most common. Leaf Switches with 10Gb downlinks have 40Gb uplink, 25Gb downlink/100Gb uplink, and 100Gb downlink/400Gb uplink models. Spine switches commonly have 32 ports matching the leaf uplink port bandwidth such as 40Gb, 100Gb, or 400Gb.

For optimal network performance, hosts are connected via dual 10|40|100Gb uplinks to the access/leaf switch layer, which in turn is connected via 40|100|400Gb links to the aggregation/spine layer. The number of servers in a two-tier Clos architecture can be determined by multiplying leaf ports per switch by ports per spine switch and dividing that number by two as each server will have 2 connections. 48 * 32/2 = 768 for example with 48 port leaf switches and 32 port spine switches.

Scaling out the architecture involves adding more hosts to the access switch pairs, and then adding more access switches in pairs or more as needed. In a Layer 2-only environment, additional spine switches should be added in pairs. Once the limit for the spine switches approaches, an additional network pod of spine/leaf switches may be added.

L2 use of MLAG/virtual port channel can increase oversubscription. 768 hosts can be connected in the pod before expansion is required. Common oversubscription ratios are formed starting from the original 4:1 uplink:downlink speed ratio based on port count usage and layer 2 MLAG versus layer 3 solutions for host connectivity redundancy.

FIGURE 47

Table bandwidth numbers are in Gbps. Leaf and Spine required throughput is calculated by totaling the interface bandwidth. Numbers and ratios are shown for both the higher and lower bandwidth usage of the uplinks. Utilizing the 100Gb port as the lower 40Gb capability is an option, but increases the oversubscription ratios. When utilizing matching uplink and downlink speeds the oversubscription can explode to unwelcome levels especially when used in conjunction with MLAG. This could be used in a migration scenario where leafs are replaced first and utilize the uplinks at the lower speeds until the spines are replaced.

48 x 10Gb 40Gb * 4 40Gb * 6 40Gb * 8 10Gb * 4 10Gb * 6 10Gb * 8 Downlink Uplinks Uplinks Uplinks Uplinks Uplinks Uplinks

Downlink 480 480 480 480 480 480 Bandwidth

Uplink 160 240 320 40 60 80 Bandwidth

Oversubscription 3.0 2.0 1.5 12.0 8.0 6.0 Ratio

MLAG Uplink 80 160 240 20 40 60 Bandwidth

MLAG 6 3 2 24.0 12.0 8.0 Oversubscription

Leaf Throughput 1280 1440 1600 2720 2880 3040 Required

Spine 2560 2560 2560 640 640 640 Throughput Required

48 x 25Gb 100Gb * 4 100Gb * 6 100Gb * 8 40Gb * 4 40Gb * 6 40Gb * 8 Downlink Uplinks Uplinks Uplinks Uplinks Uplinks Uplinks

Downlink 1200 1200 1200 1200 1200 1200 Bandwidth

Uplink 400 600 800 160 240 320 Bandwidth

Oversubscription 3.0 2.0 1.5 7.5 5.0 3.75 Ratio

MLAG Uplink 200 400 600 80 160 240 Bandwidth

MLAG 6.0 3.0 2.0 15.0 7.5 5.0 Oversubscription

Leaf Throughput 3200 3600 4000 2720 2880 3040 Required

Spine 6400 6400 6400 2560 2560 2560 Throughput Required

In practice, it is common not to use all 48 ports on leaf switches, which changes the oversubscription ratios. The chart shows numbers based on using all ports. For example, if 40 ports are used instead (common) oversubscription ratios drop to 2.5, 1.67, 1.25, 10.0, 6.67, and 5.0 for the top chart.

High performance leafs (nonblocking)

In a high performance leaf design a switch with all high bandwidth ports are used, and the bandwidth is split evenly between host ports and network ports to always form a 1:1 subscription ration between uplink and downlink bandwidth. With a one to one subscription ratio the network is nonblocking as all traffic can flow at capacity.

FIGURE 48

In the pictured example there are 32 x 100Gb ports, with 16 host ports and 16 network ports. The network ports provide uplinks and the host ports provide downlink connections to servers. The host ports are typically broken out so that there are 64 x 25Gb ports.

Troubleshooting

Describe basic troubleshooting techniques Basic steps

·· Isolate the problem

·· Implementing a fix

·· Verifying the fix resolves the problem Isolate the problem

Once a problem is reported, we need to determine the cause of the problem, and if this a problem actually caused by network related malfunction.

Traditionally this is the most time-consuming step, because it involved going device by device to find the location of the problem. Not every problem you find during troubleshooting is the problem you are trying to solve. Engineers had to refer to diagrams in large netwo rks and investigate for a lengthy period of time before the problem was isolated.

NetQ can perform the bulk of troubleshooting from a single device or management server in seconds. Any NetQ command can be executed from any Cumulus NetQ device.

cumulus@oob-mgmt-server:~$ netq check bgp Total Nodes: 10, Failed Nodes: 3, Total Sessions: 16 , Failed Sessions: 4, Hostname VRF Peer Name Peer Hostname Reason Last Changed ------leaf01 default swp51 spine01 Link Admin Down 0.138777s leaf01 default swp52 spine02 Link Admin Down 0.138839s spine01 default swp1 leaf01 Hold Timer Expired 0.138882s spine02 default swp1 leaf01 Hold Timer Expired 0.138923s

cumulus@oob-mgmt-server:~$ netq show bgp

Matching bgp records: Hostname Neighbor VRF ASN Peer ASN PfxRx Last Changed ------leaf01 swp51 default 65011 - NotEstd 0.57137s leaf01 swp52 default 65011 - NotEstd 0.57198s spine01 swp1 default 65020 - NotEstd 0.57242s spine02 swp1 default 65020 - NotEstd 0.57598s leaf02 swp51(spine01) default 65012 65020 4/-/30 52m:36.247s leaf02 swp52(spine02) default 65012 65020 4/-/30 52m:54.247s leaf03 swp51(spine01) default 65013 65020 5/-/12 52m:40.247s leaf03 swp52(spine02) default 65013 65020 5/-/12 52m:40.247s leaf04 swp51(spine01) default 65014 65020 4/-/12 52m:39.247s leaf04 swp52(spine02) default 65014 65020 4/-/12 52m:52.247s spine01 swp2(leaf02) default 65020 65012 2/-/12 52m:36.247s spine01 swp3(leaf03) default 65020 65013 2/-/17 52m:40.247s spine01 swp4(leaf04) default 65020 65014 2/-/13 52m:39.247s spine02 swp2(leaf02) default 65020 65012 2/-/12 52m:54.247s spine02 swp3(leaf03) default 65020 65013 2/-/17 52m:40.247s spine02 swp4(leaf04) default 65020 65014 2/-/13 52m:52.247s

NCLU can locally provide this information on a per device basis. In the above example an administrator shut down 2 connections bringing BGP down as shown in the netq check bgp command output. Implementing a fix

Once you’ve isolated the issue, you can take corrective action at the appropriate time based on network impact. Continuing with the previous example, the last committed change can be rolled back via the net rollback last command on device leaf01.

cumulus@leaf01:mgmt-vrf:~$ net rollback last

Verifying the fix resolves the problem

Verifying will depend on the nature of the issue and the corrective action taken, but NetQ makes it easy to validate, much in the same manner it was easy to check the infrastructure to isolate the problem.

With as previous example we run show and check bgp again to validate the corrective action. BGP is checked again via NetQ in this example focusing on leaf01 with grep, and the netq check bgp output now shows no failed sessions.

cumulus@oob-mgmt-server:~$ netq show bgp |grep leaf01

Matching bgp records: Hostname Neighbor VRF ASN Peer ASN PfxRx Last Changed ------leaf01 swp51(spine01) default 65011 65020 6/-/30 1m:41.471s leaf01 swp52(spine02) default 65011 65020 6/-/30 1m:45.472s spine01 swp1(leaf01) default 65020 65011 2/-/12 1m:42.472s spine02 swp1(leaf01) default 65020 65011 2/-/12 1m:46.472s

cumulus@oob-mgmt-server:~$ netq check bgp Total Nodes: 10, Failed Nodes: 0, Total Sessions: 16, Failed Sessions: 0

Validate layer 1 via NetQ & NCLU methods https://docs.cumulusnetworks.com/display/DOCS/Troubleshooting+Network+Interfaces

Verify link state

Net show interface

cu mulus@leaf01:m g mt-vrf:~$ net show interface State Name Spd MTU Mode LLDP Summary ------UP lo N/A 65536 Loopback IP: 127.0.0.1/8 lo IP: 10.0.0.11/32 lo IP: 10.0.0.112/32 lo IP: ::1/128 UP eth0 1G 1500 Mgmt Master: mgmt(UP) eth0 IP: 192.168.0.11/16(DHCP) UP swp1 1G 9000 BondMember server01 (eth1) Master: bond01(UP) UP swp2 1G 9000 BondMember server02 (eth1) Master: bond02(UP) UP swp49 1G 9000 BondMember leaf02 (swp49) Master: peerlink(UP) UP swp50 1G 9000 BondMember leaf02 (swp50) Master: peerlink(UP) UP swp51 1G 9216 NotConfigured spine01 (swp1) UP swp52 1G 9216 NotConfigured spine02 (swp1) UP bond01 1G 9000 802.3ad Master: bridge(UP) bond01 Bond Members: swp1(UP) UP bond02 1G 9000 802.3ad Master: bridge(UP) bond02 Bond Members: swp2(UP) UP bridge N/A 9000 Bridge/L2 UP mgmt N/A 65536 Interface/L3 IP: 127.0.0.1/8 UP peerlink 2G 9000 802.3ad Master: bridge(UP)

peerlink Bond Members: swp49(UP) peerlink Bond Members: swp50(UP) UP peerlink.4094 2G 9000 SubInt/L3 IP: 169.254.1.1/30 UP vlan13 N/A 9000 Interface/L3 IP: 10.1.3.11/24 UP vlan13-v0 N/A 9000 Interface/L3 IP: 10.1.3.1/24 UP vlan24 N/A 9000 Interface/L3 IP: 10.2.4.11/24 UP vlan24-v0 N/A 9000 Interface/L3 IP: 10.2.4.1/24 UP vni13 N/A 9000 Access/L2 Master: bridge(UP) UP vni24 N/A 9000 Access/L2 Master: bridge(UP)

NetQ show interfaces (specific to leaf01)

cumulus@oob-mgmt-server:~$ netq leaf01 show interfaces

Matching link records: Hostname Interface Type State VRF Details Last Changed ------leaf01 bond01 bond up default Slave: swp1(server01:eth1), 5m:22.692s VLANs: , PVID: 13, Master: bridge, MTU: 9000 leaf01 bond02 bond up default Slave: swp2(server02:eth1), 5m:22.686s VLANs: , PVID: 24, Master: bridge, MTU: 9000 leaf01 bridge bridge up default Members:bond02,vni13,peerlink,bond 5m:22.684s 01,vni24, Root Bridge: leaf01, Root port N/A, MTU: 9000 leaf01 eth0 eth up mgmt MTU:1500 5m:24.695s leaf01 lo loopback up default MTU:65536 5m:48.733s leaf01 mgmt vrf up table: 1001, MTU: 65536, Members: 14m:52.979s leaf01 peerlink bond up default Slave: swp50(leaf02:swp50), 5m:22.699s Slave: swp49(leaf02:swp49), VLANs: 13 24, PVID: 1, Master: bridge, MTU: 9000 leaf01 peerlink.4094 vlan up default MTU:9000 5m:46.959s leaf01 swp1 swp up default LLDP: server01:eth1, MTU: 9000 5m:24.682s leaf01 swp2 swp up default LLDP: server02:eth1, MTU: 9000 5m:24.680s leaf01 swp44 swp down default MTU:1500 1h:1m:41s leaf01 swp45 swp down default MTU:1500 1h:1m:41s leaf01 swp46 swp down default MTU:1500 1h:1m:41s leaf01 swp47 swp down default MTU:1500 1h:1m:41s leaf01 swp48 swp down default MTU:1500 1h:1m:41s leaf01 swp49 swp up default LLDP: leaf02:swp49, MTU: 9000 5m:24.685s

leaf01 swp50 swp up default LLDP: leaf02:swp50, MTU: 9000 5m:24.684s leaf01 swp51 swp up default LLDP: spine01:swp1, MTU: 9216 5m:48.812s leaf01 swp52 swp up default LLDP: spine02:swp1, MTU: 9216 5m:48.761s leaf01 vlan13 vlan up default MTU:9000 5m:41.885s leaf01 vlan13-v0 macvlan up default MAC: 44:39:39:ff:00:13, 5m:41.828s Mode: Private leaf01 vlan24 vlan up default MTU:9000 5m:41.842s leaf01 vlan24-v0 macvlan up default MAC: 44:39:39:ff:00:24, 5m:41.816s Mode: Private leaf01 vni13 vxlan up default VNI: 13, PVID: 13, Master: bridge, 5m:22.696s VTEP: 10.0.0.112, MTU: 9000 leaf01 vni24 vxlan up default VNI: 24, PVID: 24, Master: bridge, 5m:22.690s VTEP: 10.0.0.112, MTU: 9000

NetQ check interfaces

cumulus@oob-mgmt-server:~$ netq check interfaces Checked Nodes: 10, Failed Nodes: 0 Checked Ports: 72, Failed Ports: 0, Unverified Ports: 24

Net show clag

cu mulus@leaf01:m g mt-vrf:~$ net show clag The peer is alive Our Priority, ID, and Role: 100 44:38:39:00:02:03 primary Peer Priority, ID, and Role: 200 44:38:39:00:03:03 secondary Peer Interface and IP: peerlink.4094 169.254.1.2 VxLAN Anycast IP: 10.0.0.112 Backup IP: 10.0.0.12 (active) System MAC: 44:39:39:ff:40:94

CLAG Interfaces Our Interface Peer Interface CLAG Id Conflicts Proto-Down Reason ------vni13 vni13 - - - bond01 bond01 1 - - vni24 vni24 - - - bond02 bond02 2 - -

cumulus@leaf01:mgmt-vrf:~$ net show clag neighbors 10.1.3.101 vlan13 44:38:39:00:08:01 13 local,peer 10.2.4.102 vlan24 44:38:39:00:09:01 24 local,peer

NetQ show clag

cumulus@oob-mgmt-server:~$ netq show clag

Matching clag records: Hostname Peer SysMac State Backup #Bond #Dual Last Changed ------leaf01(P) eaf02 44:39:39:ff:40:94 up up 4 4 22m:23.293s leaf02 leaf01(P) 44:39:39:ff:40:94 up up 4 4 22m:23.804s leaf03(P) leaf04 44:39:39:ff:40:95 up up 4 4 22m:23.234s leaf04 leaf03(P) 44:39:39:ff:40:95 up up 4 4 22m:24.928s

NetQ check clag

cumulus@oob-mgmt-server:~$ netq check clag Checked Nodes: 4, Failed Nodes: 0

If configured Prescriptive Topology Manager (PTM) can be used.

https://docs.cumulusnetworks.com/display/DOCS/Prescriptive+Topology+Manager+-+PTM

Verify counters

Net show counters

cu mulus@leaf01:m g mt-vrf:~$ net show counters

Kernel Interface table Iface MTU Met RX _ OK RX _ ERR RX _ DRP RX _ OVR TX _ OK TX _ ERR TX _ DRP TX _ OVR Flg ------bond01 9000 0 9222 0 0 0 13390 0 0 0 BMmRU bond02 9000 0 9194 0 0 0 13498 0 0 0 BMmRU bridge 9000 0 852 0 0 0 217 0 0 0 BMRU eth0 1500 0 16087 0 0 0 18255 0 0 0 BMRU lo 65536 0 1316 0 0 0 1316 0 0 0 LRU mgmt 65536 0 17576 0 0 0 19156 0 0 0 OmRU

peerlink 9000 0 27237 0 0 0 27108 0 0 0 BMmRU peerlink.4094 9000 0 8892 0 0 0 8892 0 0 0 BMRU swp1 9000 0 9243 0 9 0 13390 0 0 0 BMPsRU swp2 9000 0 9222 0 9 0 13498 0 0 0 BMPsRU swp49 9000 0 9369 0 0 0 9250 0 0 0 BMPsRU swp50 9000 0 17868 0 0 0 17858 0 0 0 BMPsRU swp51 9216 0 6789 0 1 0 4662 0 0 0 BMRU swp52 9216 0 10946 0 1 0 9670 0 0 0 BMRU vlan13 9000 0 531 0 0 0 105 0 0 0 BMRU vlan24 9000 0 321 0 0 0 106 0 0 0 BMRU vlan13-v0 9000 0 277 0 0 0 64 0 0 0 BMRU vlan24-v0 9000 0 92 0 0 0 64 0 0 0 BMRU vni13 9000 0 0 0 0 0 283 2 0 2 BMRU vni24 9000 0 0 0 0 0 163 0 0 0 BMRU

Verify bonding

Net show interface bonds

cu mulus@leaf01:m g mt-vrf:~$ net show interface bonds mac Name MAC Speed MTU Mode Summary ------UP bond01 44:38:39:00:02:05 1G 9000 802.3ad Bond Members: swp1(UP) UP bond02 44:38:39:00:02:06 1G 9000 802.3ad Bond Members: swp2(UP) UP peerlink 44:38:39:00:02:03 2G 9000 802.3ad Bond Members: swp49(UP), swp50(UP)

Net show interface bondmems mac

cumulus@leaf01:mgmt-vrf:~$ net show interface bondmems mac Name MAC Speed MTU Mode Summary ------UP swp1 44:38:39:00:02:05 1G 9000 LACP-UP Master: bond01(UP) UP swp2 44:38:39:00:02:06 1G 9000 LACP-UP Master: bond02(UP) UP swp49 44:38:39:00:02:03 1G 9000 LACP-UP Master: peerlink(UP) UP swp50 44:38:39:00:02:03 1G 9000 LACP-UP Master: peerlink(UP)

NetQ show interfaces type bond

cu mulus@leaf01:m g mt-vrf:~$ netq leaf04 show interfaces type bond

Matching link records: Hostname Interface Type State VRF Details Last Changed ------leaf04 bond03 bond up default Slave: swp1(server03:eth2), 3h:8m:26s VLANs: , PVID:13, Master: bridge, MTU: 9000 leaf04 bond04 bond up default Slave: swp2(server04:eth2), 3h:8m:26s VLANs: , PVID:24, Master: bridge, MTU: 9000 leaf04 peerlink bond up default Slave: swp50(leaf03:swp50), 3h:8m:26s Slave: swp49(leaf03:swp49), VLANs: 13 24, PVID: 1, Master: bridge, MTU: 9000

Validate layer 2 via NetQ & NCLU methods Verify spanning tree

Net show bridge spanning-tree detail

cu mulus@leaf01:m g mt-vrf:~$ net show bridge spanning-tree Bridge info enabled yes bridge id 8.000.44:39:39:FF:40:94 Priority: 32768 Address: 44:39:39:FF:40:94 This bridge is root.

designated root 8.000.44:39:39:FF:40:94 Priority: 32768 Address: 44:39:39:FF:40:94

root port none path cost 0 internal path cost 0 max age 20 bridge max age 20 forward delay 15 bridge forward delay 15 tx hold count 6 max hops 20 hello time 2 ageing time 300 force protocol version rstp

E bond01 8.001 forw 8.000.44:39:39:FF:40:94 8.000.44:39:39:FF:40:94 8.001 Desg E bond02 8.002 forw 8.000.44:39:39:FF:40:94 8.000.44:39:39:FF:40:94 8.002 Desg

E peerlink F.003 forw 8.000.44:39:39:FF:40:94 8.000.44:39:39:FF:40:94 F.003 Desg E vni13 8.004 forw 8.000.44:39:39:FF:40:94 8.000.44:39:39:FF:40:94 8.004 Desg E vni24 8.005 forw 8.000.44:39:39:FF:40:94 8.000.44:39:39:FF:40:94 8.005 Desg

Verify VLANs

Net show bridge VLAN

cu mulus@leaf01:m g mt-vrf:~$ net show bridge vlan 13

Interface VLAN Flags VNI ------bond01 13 PVID, Egress Untagged bridge 13 peerlink 13 vni13 13 PVID, Egress Untagged 13

NetQ show VLAN

cu mulus@leaf01:m g mt-vrf:~$ netq show vlan

Matching vlan records: Hostname VLANs SVIs Last Changed ------leaf01 1,13,24 13 24 3h:19m:20s leaf02 1,13,24 13 24 3h:19m:21s leaf03 1,13,24 13 24 3h:19m:20s leaf04 1,13,24 13 24 3h:19m:20s

NetQ check VLAN

cu mulus@leaf01:m g mt-vrf:~$ netq check vlan Checked Nodes: 10, Checked Links: 140, Failed Nodes: 0, Failed Links: 0 No VLAN or PVID Mismatch found

You can view a list of configured VXLANs for all devices, including the VNI, protocol, address of associated VTEPs, replication list for BUM traffic, and the last time it was changed. VXLAN information for a given device is show by adding a hostname to the show command.

cu mulus@switch:~$ netq show vxlan Matching vxlan records: Hostname VNI Protocol VTEP IP VLAN Replication List Last Changed ------exit01 104001 EVPN 10.0.0.41 4001 22h:50m:1s exit02 104001 EVPN 10.0.0.42 4001 22h:49m:38s leaf01 13 EVPN 10.0.0.112 13 10.0.0.134(leaf04, leaf03) 23h:43m:1s leaf01 24 EVPN 10.0.0.112 24 10.0.0.134(leaf04, leaf03) 23h:43m:1s leaf01 104001 EVPN 10.0.0.112 4001 23h:43m:1s leaf02 13 EVPN 10.0.0.112 13 10.0.0.134(leaf04, leaf03) 22h:56m:22s leaf02 24 EVPN 10.0.0.112 24 10.0.0.134(leaf04, leaf03) 22h:56m:22s leaf02 104001 EVPN 10.0.0.112 4001 22h:56m:22s leaf03 13 EVPN 10.0.0.134 13 10.0.0.112(leaf02, leaf01) 22h:55m:33s leaf03 24 EVPN 10.0.0.134 24 10.0.0.112(leaf02, leaf01) 22h:55m:33s leaf03 104001 EVPN 10.0.0.134 4001 22h:55m:33s leaf04 13 EVPN 10.0.0.134 13 10.0.0.112(leaf02, leaf01) 22h:52m:12s leaf04 24 EVPN 10.0.0.134 24 10.0.0.112(leaf02, leaf01) 22h:52m:12s leaf04 104001 EVPN 10.0.0.134 4001 22h:52m:12s

cu mulus@switch:~$ netq leaf02 show interfaces type vxlan Matching link records: Hostname Interface Type State VRF Details Last Changed ------leaf02 vni13 vxlan up default VNI: 13, PVID: 13, Master: bridge, 23h:23m:11s VTEP: 10.0.0.112, MTU: 9000 leaf02 vni24 vxlan up default VNI: 24, PVID: 24, Master: bridge, 23h:23m:11s VTEP: 10.0.0.112, MTU: 9000 leaf02 vxlan4001 vxlan up default VNI: 104001, PVID: 4001, 23h:23m:11s Master: bridge, VTEP: 10.0.0.112, MTU: 1500

NetQ check VXLAN

cu mulus@leaf01:m g mt-vrf:~$ netq check vxlan Checked Nodes: 4, Warning Nodes: 0, Failed Nodes: 0

Validate layer 3 via NetQ & NCLU methods IP addressing

NetQ show IP addresses

cu mulus@leaf01:m g mt-vrf:~$ netq show ip addresses vlan13

Matching address records: Address Hostname Interface VRF Last Changed ------10.1.3.1/24 leaf01 vlan13-v0 default 3h:25m:15s 10.1.3.1/24 leaf02 vlan13-v0 default 3h:25m:16s 10.1.3.1/24 leaf03 vlan13-v0 default 3h:25m:16s 10.1.3.1/24 leaf04 vlan13-v0 default 3h:25m:16s 10.1.3.11/24 leaf01 vlan13 default 3h:25m:15s 10.1.3.12/24 leaf02 vlan13 default 3h:25m:16s 10.1.3.13/24 leaf03 vlan13 default 3h:25m:16s 10.1.3.14/24 leaf04 vlan13 default 3h:25m:16s fe80::4639:39ff:feff:13/64 leaf01 vlan13-v0 default 3h:25m:15s fe80::4639:39ff:feff:13/64 leaf02 vlan13-v0 default 3h:25m:16s fe80::4639:39ff:feff:13/64 leaf03 vlan13-v0 default 3h:25m:16s fe80::4639:39ff:feff:13/64 leaf04 vlan13-v0 default 3h:25m:16s

NetQ show IPv6 addresses

cu mulus@leaf01:m g mt-vrf:~$ netq spine01 show ipv6 addresses

Matching address records: Address Hostname Interface VRF Last Changed ------fe80::4638:39ff:fe00:600/64 spine01 eth0 mgmt 3h:38m:1s fe80::4638:39ff:fe00:601/64 spine01 swp1 default 3h:32m:7s fe80::4638:39ff:fe00:602/64 spine01 swp2 default 3h:32m:7s fe80::4638:39ff:fe00:603/64 spine01 swp3 default 3h:32m:7s fe80::4638:39ff:fe00:604/64 spine01 swp4 default 3h:32m:6s fe80::4638:39ff:fe00:605/64 spine01 swp31 default 3h:32m:6s fe80::4638:39ff:fe00:606/64 spine01 swp32 default 3h:32m:7s

Route peering

Refer to BGP Unnumbered Troubleshooting earlier in this document for more BGP examples and show command output.

Net show OSPF

cu mulus@spine01:m g mt-vrf:~$ net show ospf neighbor Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL 10.0.0.11 1 Full/DROther 33.973s 10.0.0.11 swp1:10.0.0.21 0 0 0

cumulus@spine01:mgmt-vrf:~$ net show ospf neighbor detail Neighbor 10.0.0.11, interface address 10.0.0.11 In the area 0.0.0.0 via interface swp1 Neighbor priority is 1, State is Full, 5 state changes Most recent state change statistics: Progressive change 1m09s ago DR is 0.0.0.0, BDR is 0.0.0.0 O ptions 2 *|-|-|-|-|-|E|- Dead timer due in 30.743s Database Summary List 0 Link State Request List 0 Link State Retransmission List 0 Thread Inactivity Timer on Thread Database Description Retransmision off Thread Link State Request Retransmission on Thread Link State Update Retransmission on

NetQ show OSPF

cumulus@oob-mgmt-server:~$ netq show ospf

Matching ospf records: Hostname Interface Area Type State PeerHostname PeerInterface LastChanged ------leaf01 swp51 0.0.0.0 Unnumbered Full spine01 swp1 1m:53.293s spine01 swp1 0.0.0.0 Unnumbered Full leaf01 swp51 1m:52.857s

NetQ check OSPF

cumulus@oob-mgmt-server:~$ netq check ospf Total Sessions: 2, Failed Sessions: 0

Net show BGP

Network Next Hop Metric LocPrf Weight Path *> 10.0.0.11/32 0.0.0.0 0 32768 i *= 10.0.0.12/32 swp51 0 65020 65012 i *> swp52 0 65020 65012 i *= 10.0.0.13/32 swp51 0 65020 65013 i *> swp52 0 65020 65013 i *= 10.0.0.14/32 swp51 0 65020 65014 i *> swp52 0 65020 65014 i *> 10.0.0.21/32 swp51 0 0 65020 i *> 10.0.0.22/32 swp52 0 0 65020 i * 10.0.0.112/32 swp51 0 65020 65012 i * swp52 0 65020 65012 i *> 0.0.0.0 0 32768 i *= 10.0.0.134/32 swp52 0 65020 65014 i *> swp51 0 65020 65014 i

Displayed 8 routes and 14 total paths

show bgp ipv6 unicast ======No BGP prefixes displayed, 0 exist

Net show BGP summary

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd spine01(swp51) 4 65020 4362 4361 0 0 0 03:34:57 6 spine02(swp52) 4 65020 4362 4361 0 0 0 03:34:57 6

Total number of neighbors 2

show bgp ipv6 unicast summary ======% No BGP neighbors found

show bgp l2vpn evpn summary ======BGP router identifier 10.0.0.11, local AS number 65011 vrf-id 0 BGP table version 0 RIB entries 15, using 2280 bytes of memory Peers 2, using 39 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd spine01(swp51) 4 65020 4362 4361 0 0 0 03:34:58 28 spine02(swp52) 4 65020 4363 4362 0 0 0 03:34:58 28

Total number of neighbors 2

NetQ show BGP

cu mulus@leaf01:m g mt-vrf:~$ netq show bgp

Matching bgp records: Hostname Neighbor VRF ASN Peer ASN PfxRx Last Changed ------leaf01 swp51(spine01) default 65011 65020 6/-/28 3h:36m:29s leaf01 swp52(spine02) default 65011 65020 6/-/28 3h:36m:29s leaf02 swp51(spine01) default 65012 65020 5/-/28 3h:36m:28s leaf02 swp52(spine02) default 65012 65020 5/-/28 3h:36m:29s leaf03 swp51(spine01) default 65013 65020 6/-/28 3h:36m:30s leaf03 swp52(spine02) default 65013 65020 6/-/28 3h:36m:30s leaf04 swp51(spine01) default 65014 65020 5/-/28 3h:36m:29s leaf04 swp52(spine02) default 65014 65020 5/-/28 3h:36m:29s spine01 swp1(leaf01) default 65020 65011 2/-/12 3h:36m:29s spine01 swp2(leaf02) default 65020 65012 2/-/16 3h:36m:29s spine01 swp3(leaf03 default 65020 65013 2/-/12 3h:36m:29s spine01 swp4(leaf04) default 65020 65014 2/-/16 3h:36m:29s

spine02 swp1(leaf01) default 65020 65011 2/-/12 3h:36m:30s spine02 swp2(leaf02) default 65020 65012 2/-/16 3h:36m:30s spine02 swp3(leaf03) default 65020 65013 2/-/12 3h:36m:30s spine02 swp4(leaf04) default 65020 65014 2/-/16 3h:36m:30

NetQ check BGP

cu mulus@leaf01:m g mt-vrf:~$ netq check bgp Total Nodes: 10, Failed Nodes: 0, Total Sessions: 16, Failed Sessions: 0

cumulus@oob-mgmt-server:~$ netq check bgp Total Nodes: 10, Failed Nodes: 2, Total Sessions: 16 , Failed Sessions: 2, Hostname PeerName PeerHostname Reason Last Changed ------leaf04 swp51 spine01 evpn address families not activated on peer 34.140580s spine01 swp4 leaf04 evpn address families not activated on node 33.140629s

Route Table

Net show route

cu mulus@leaf01:m g mt-vrf:~$ net show route show ip route ======Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, > - selected route, * - FIB route

C>* 10.0.0.11/32 is directly connected, lo, 03:42:36 B>* 10.0.0.12/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 03:42:31 * via fe80::4638:39ff:fe00:601, swp51, 03:42:31 B>* 10.0.0.13/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 03:42:31 * via fe80::4638:39ff:fe00:601, swp51, 03:42:31 B>* 10.0.0.14/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 03:42:31 * via fe80::4638:39ff:fe00:601, swp51, 03:42:31 B>* 10.0.0.21/32 [20/0] via fe80::4638:39ff:fe00:601, swp51, 03:42:31 B>* 10.0.0.22/32 [20/0] via fe80::4638:39ff:fe00:701, swp52, 03:42:31 C>* 10.0.0.112/32 is directly connected, lo, 03:42:25 B>* 10.0.0.134/32 [20/0] via fe80::4638:39ff:fe00:601, swp51, 03:42:26

* via fe80::4638:39ff:fe00:701, swp52, 03:42:26 C * 10.1.3.0/24 is directly connected, vlan13-v0, 03:42:33 C>* 10.1.3.0/24 is directly connected, vlan13, 03:42:33 C * 10.2.4.0/24 is directly connected, vlan24-v0, 03:42:33 C>* 10.2.4.0/24 is directly connected, vlan24, 03:42:33 C>* 169.254.1.0/30 is directly connected, peerlink.4094, 03:42:35

show ipv6 route ======Codes: K - kernel route, C - connected, S - static, R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, > - selected route, * - FIB route

C * fe80::/64 is directly connected, peerlink.4094, 03:42:33 C * fe80::/64 is directly connected, vlan24-v0, 03:42:33 C * fe80::/64 is directly connected, vlan13-v0, 03:42:33 C * fe80::/64 is directly connected, vlan24, 03:42:33 C * fe80::/64 is directly connected, vlan13, 03:42:33 C * fe80::/64 is directly connected, bridge, 03:42:33 C * fe80::/64 is directly connected, swp52, 03:42:35 C>* fe80::/64 is directly connected, swp51, 03:42:35

Net show route summary

cu mulus@leaf01:m g mt-vrf:~$ net show route summary show ip route summary ======Route Source Routes FIB (vrf default) connected 7 5 ebgp 6 6 ibgp 0 0 ------Totals 13 11

show ipv6 route summary ======Route Source Routes FIB (vrf default) connected 8 1 ------Totals 8 1

EVPN

Net show EVPN

cu mulus@leaf01:m g mt-vrf:~$ net show evpn L2 VNIs: 2 L3 VNIs: 0 Advertise gateway mac-ip: No Duplicate address detection: Enable Detection max-moves 5, time 180

NetQ show EVPN

cu mulus@leaf01:m g mt-vrf:~$ netq show evpn

Matching evpn records: Hostname VNI VTEP IP In Kernel Export RT Import RT Last Changed ------leaf01 13 10.0.0.112 yes 65011:13 65011:13 3h:51m:28s leaf01 24 10.0.0.112 yes 65011:24 65011:24 3h:51m:28s leaf02 13 10.0.0.112 yes 65012:13 65012:13 3h:51m:27s leaf02 24 10.0.0.112 yes 65012:24 65012:24 3h:51m:27s leaf03 13 10.0.0.134 yes 65013:13 65013:13 3h:51m:28s leaf03 24 10.0.0.134 yes 65013:24 65013:24 3h:51m:28s leaf04 13 10.0.0.134 yes 65014:13 65014:13 3h:51m:29s leaf04 24 10.0.0.134 yes 65014:24 65014:24 3h:51m:29s

NetQ check EVPN

cu mulus@leaf01:m g mt-vrf:~$ netq check evpn Total Nodes:10, Failed Nodes:0, Total Sessions:8, Failed Sessions:0, Total VNIs:2

Net show BGP EVPN route

cu mulus@leaf01:m g mt-vrf:~$ net show bgp evpn route BGP table version is 10, local router ID is 10.0.0.11 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete E V P N t y p e -2 p r e fi x: [2]:[E SI]:[E t h T a g]:[ M A C l e n]:[ M A C]:[I Pl e n]:[I P]

E V P N t y p e -3 p r e fi x: [3]:[E t h T a g]:[I Pl e n]:[O r i gI P] E V P N t y p e -5 p r e fi x: [5]:[E SI]:[E t h T a g]:[I Pl e n]:[I P]

Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 10.0.0.11:2 * > [2]:[0]:[0]:[48]:[44:38:39:0 0:03:05] 10.0.0.112 32768 i * > [2]:[0]:[0]:[48]:[44:38:39:0 0:08:01] 10.0.0.112 32768 i * > [2]:[0]:[0]:[48]:[44:38:39:0 0:08:01]:[32]:[10.1.3.101] 10.0.0.112 32768 i * > [2]:[0]:[0]:[48]:[46:38:39:0 0:08:01] 10.0.0.112 32768 i * > [2]:[0]:[0]:[48]:[46:38:39:0 0:08:02] 10.0.0.112 32768 i * > [3]:[0]:[32]:[10.0.0.11 2] 10.0.0.112 32768 i Route Distinguisher: 10.0.0.13:2 * [2]:[0]:[0]:[48]:[44:38:39:0 0:05:05] 10.0.0.134 0 65020 65013 i * > [2]:[0]:[0]:[48]:[44:38:39:0 0:05:05] 10.0.0.134 0 65020 65013 i * > [2]:[0]:[0]:[48]:[44:38:39:0 0:0 a:01] 10.0.0.134 0 65020 65013 i * [2]:[0]:[0]:[48]:[44:38:39:0 0:0 a:01] 10.0.0.134 0 65020 65013 i * > [2]:[0]:[0]:[48]:[44:38:39:0 0:0 a:01]:[32]:[10.1.3.103] 10.0.0.134 0 65020 65013 i * [2]:[0]:[0]:[48]:[44:38:39:0 0:0 a:01]:[32]:[10.1.3.103] 10.0.0.134 0 65020 65013 i * > [2]:[0]:[0]:[48]:[46:38:39:0 0:0 a:01] 10.0.0.134 0 65020 65013 i * [2]:[0]:[0]:[48]:[46:38:39:0 0:0 a:01] 10.0.0.134 0 65020 65013 i * > [2]:[0]:[0]:[48]:[46:38:39:0 0:0 a:02] 10.0.0.134 0 65020 65013 i * [2]:[0]:[0]:[48]:[46:38:39:0 0:0 a:02] 10.0.0.134 0 65020 65013 i * [3]:[0]:[32]:[10.0.0.134] 10.0.0.134 0 65020 65013 i * > [3]:[0]:[32]:[10.0.0.134] 10.0.0.134 0 65020 65013 i

Net show BGP EVPN summary

cu mulus@leaf01:m g mt-vrf:~$ net show bgp evpn summary json : Print output in json cumulus@leaf01:mgmt-vrf:~$ net show bgp evpn summary BGP router identifier 10.0.0.11, local AS number 65011 vrf-id 0 BGP table version 0 RIB entries 15, using 2280 bytes of memory Peers 2, using 39 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd spine01(swp51) 4 65020 4878 4877 0 0 0 04:00:46 28 spine02(swp52) 4 65020 4879 4878 0 0 0 04:00:46 28

Total number of neighbors 2

Net show BGP EVPN VNI

cu mulus@leaf01:m g mt-vrf:~$ net show bgp evpn vni Advertise Gateway Macip: Disabled Advertise All VNI flag: Enabled BUM flooding: Head-end replication Number of L2 VNIs: 2 Number of L3 VNIs: 0 Flags: * - Kernel VNI Type RD Import RT Export RT Tenant VRF * 24 L2 10.0.0.11:3 65011:24 65011:24 default * 13 L2 10.0.0.11:2 65011:13 65011:13 default

Net show EVPN ARP-cache VNI 24

cu mulus@leaf01:m g mt-vrf:~$ net show evpn arp-cache vni 24 Number of ARPs (local and remote) known for this VNI: 8 IP Type State MAC Remote VTEP fe80::4638:39ff:fe00:205 local active 44:38:39:00:02:05 10.2.4.104 remote active 44:38:39:00:0b:01 10.0.0.134 10.2.4.102 local active 44:38:39:00:09:01 10.2.4.13 remote active 44:38:39:00:04:05 10.0.0.134 10.2.4.1 local active 44:39:39:ff:00:24

fe80::4638:39ff:fe00:405 remote active 44:38:39:00:04:05 10.0.0.134 fe80::4639:39ff:feff:24 local active 44:39:39:ff:00:24 10.2.4.11 local active 44:38:39:00:02:05

NetQ path trace https://docs.cumulusnetworks.com/display/NETQ/Monitor+Network+Layer+Protocols#MonitorNetworkLay erProtocols-ViewPathsbetweenDevices

You can view the available unidirectional or bidirectional paths between two devices on the network currently and historically using their IPv4 or IPv6 addresses, and view the output in json, pretty, or detail format.

·· JSON output provides the output in a JSON file format for ease of importing to other applications or software.

·· Pretty output lines up the paths in a pseudo-graphical manner to help visualize multiple paths.

·· Detail output is the default when not specified and is useful for traces with higher hop counts where the pretty output wraps lines, making it harder to interpret the results. The detail output displays a table with a row per hop and a set of rows per path.

netq trace from (|) [vrf ] [around ] [bidir] [json|detail|pretty] [debug]

cu mulus@switch:~$ netq trace 10.0.0.13 from 10.0.0.11 bidir pretty Number of Paths: 2 Number of Paths with Errors: 0 Number of Paths with Warnings: 0 Path MTU: 1500

leaf01 swp52 -- swp1 spine02 swp3 -- swp52 leaf03 swp51 -- swp1 spine01 swp3 -- swp51 leaf03

Number of Paths: 2 Number of Paths with Errors: 0 Number of Paths with Warnings: 0 Path MTU: 1500

leaf03 swp52 -- swp3 spine02 swp1 -- swp52 leaf01 swp51 -- swp3 spine01 swp1 -- swp51 leaf01

Linux system validation Display & validate CPU utilization

Top

cu mulus@leaf01:m g mt-vrf:~$ top top - 21:22:43 up 7:01, 2 users, load average: 0.22, 0.20, 0.18 Tasks: 116 total, 1 running, 115 sleeping, 0 stopped, 0 zombie %Cpu(s): 10.1 us, 3.0 sy, 0.0 ni, 86.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.3 st KiB Mem: 1981808 total, 487760 used, 1494048 free, 876 buffers KiB Swap: 0 total, 0 used, 0 free. 242384 cached Mem

P I D U S E R P R N I V I R T R E S S H R S % C P U % M E M T I M E + C O M M A N D 1 5 3 8 6 r o o t 1 5 - 5 7 7 6 5 1 6 2 0 6 4 4 7 6 7 2 S 2 . 0 1 . 0 8 : 0 1 . 2 0 c l a g d 1 6 0 5 5 r o o t 2 0 0 7 8 9 5 2 2 6 2 6 8 7 8 7 2 S 2 . 0 1 . 3 4 : 1 6 . 2 5 n e t q - a g e n t 1 r o o t 2 0 0 2 9 9 0 0 5 6 2 0 3 2 5 6 S 1 . 0 0 . 3 0 : 4 8 . 7 8 s y s t e m d 2 3 8 8 5 c u m u l u s 2 0 0 4 4 4 3 6 5 3 2 0 4 6 3 6 S 0 . 7 0 . 3 0 : 0 0 . 3 8 s s h 3 9 8 m e s s a g e + 2 0 0 4 2 1 2 4 3 3 1 6 2 9 4 0 S 0 . 3 0 . 2 0 : 0 2 . 5 1 d b u s - d a e m o n 6 6 8 c u m u l u s 2 0 0 2 5 7 6 4 3 0 4 4 2 4 8 4 R 0 . 3 0 . 2 0 : 0 0 . 0 1 t o p 7 6 4 r o o t 2 0 0 4 9 5 1 2 8 1 6 4 1 2 7 2 2 0 S 0 . 3 0 . 8 2 : 4 3 . 4 8 n e i g h m g r d 1 3 0 0 2 n t p 2 0 0 1 0 3 7 1 6 5 3 2 0 4 5 8 4 S 0 . 3 0 . 3 0 : 0 3 . 1 0 n t p d 2 4 0 0 5 c u m u l u s 2 0 0 8 2 2 2 0 4 4 8 8 3 6 2 4 S 0 . 3 0 . 2 0 : 0 0 . 3 5 s s h d 24248 cumulus 20 0 84292 3480 2612 S 0.3 0.2 0:00.80 sshd

IOstat

cu mulus@leaf01:m g mt-vrf:~$ iostat -xtc 5 3 Linux 4.1.0-cl-7-amd64 (leaf01) 02/21/2019 _ x86 _ 64 _ (1 CPU)

02/21/2019 07:53:43 PM avg-cpu: %user %nice %system %iowait %steal %idle 8.58 0.00 3.42 0.12 0.25 87.63

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r _ await w _ await svctm %util vda 0.00 0.00 0.00 0.00 0.02 0.00 7.69 0.00 0.58 0.58 0.00 0.57 0.00 sda 0.00 0.29 0.27 1.32 8.19 56.79 81.62 0.01 7.29 20.40 4.60 2.57 0.41

MPstat (sysstat)

cu mulus@leaf01:m g mt-vrf:~$ sudo apt-get install sysstat

cumulus@leaf01:mgmt-vrf:~$ mpstat -P ALL Linux 4.1.0-cl-7-amd64 (leaf01) 02/21/2019 _ x86 _ 64 _ (1 CPU)

07:58:17 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 07:58:17 PM all 8.58 0.01 3.34 0.12 0.00 0.08 0.24 0.00 0.00 87.63 07:58:17 PM 0 8.58 0.01 3.34 0.12 0.00 0.08 0.24 0.00 0.00 87.63

Display & validate memory utilization

Memory can be seen in the output of the top command as referenced in the CPU section, as well as the below examples.

Free

cu mulus@leaf01:m g mt-vrf:~$ free total used free shared buffers cached Mem: 1981808 492992 1488816 6188 876 244816 -/+ buffers/cache: 247300 1734508 Swap: 0 0 0

/proc/meminfo

cu mulus@leaf01:m g mt-vrf:~$ cat /proc/meminfo MemTotal: 1981808 kB MemFree: 1488012 kB MemAvailable: 1627940 kB Buffers: 876 kB Cached: 242668 kB SwapCached: 0 kB Active: 362652 kB Inactive: 59172 kB Active(anon): 179052 kB Inactive(anon): 5416 kB Active(file): 183600 kB Inactive(file): 53756 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB

SwapFree: 0 kB Dirty: 136 kB Writeback: 0 kB AnonPages: 178324 kB Mapped: 37620 kB Shmem: 6188 kB Slab: 42892 kB SReclaimable: 30392 kB SUnreclaim: 12500 kB KernelStack: 2400 kB PageTables: 6216 kB NFS _ Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 990904 kB Committed _ AS: 554544 kB VmallocTotal: 34359738367 kB VmallocUsed: 11048 kB VmallocChunk: 34359724992 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB HugePages _ Total: 0 HugePages _ Free: 0 HugePages _ Rsvd: 0 HugePages _ Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 71544 kB DirectMap2M: 2025472 kB

Display & validate disc utilization

cu mulus@leaf01:m g mt-vrf:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 10M 0 10M 0% /dev tmpfs 388M 5.8M 382M 2% /run /dev/sda4 5.8G 1.2G 4.4G 21% / tmpfs 968M 0 968M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 968M 0 968M 0% /sys/fs/cgroup /dev/sda4 5.8G 1.2G 4.4G 21% /var/lib/libvirt/images /dev/sda4 5.8G 1.2G 4.4G 21% /var/cache/cumulus /dev/sda4 5.8G 1.2G 4.4G 21% /var/opt

/dev/sda4 5.8G 1.2G 4.4G 21% /var/spool /dev/sda4 5.8G 1.2G 4.4G 21% /usr/local /dev/sda4 5.8G 1.2G 4.4G 21% /srv /dev/sda4 5.8G 1.2G 4.4G 21% /opt /dev/sda4 5.8G 1.2G 4.4G 21% /var/log /dev/sda4 5.8G 1.2G 4.4G 21% /home /dev/sda4 5.8G 1.2G 4.4G 21% /var/support /dev/sda4 5.8G 1.2G 4.4G 21% /tmp /dev/sda4 5.8G 1.2G 4.4G 21% /var/tmp /dev/sda4 5.8G 1.2G 4.4G 21% /.snapshots /dev/sda4 5.8G 1.2G 4.4G 21% /boot/grub/i386-pc /dev/sda4 5.8G 1.2G 4.4G 21% /boot/grub/x86 _ 64-efi

Display & validate services

Services can be checked directly within Linux or via NetQ. The systemctl status .service command will provide the status of an individual service. You can enter ntp, another service, or a wildcard in order to check service status.

cumulus@oob-mgmt-server:~$ systemctl status ntp.service ● ntp.service - NTP - Network Time Protocol daemon Loaded: loaded (/lib/systemd/system/ntp.service; enabled) Active: active (running) since Fri 2019-03-01 15:02:25 UTC; 2h 22min ago Docs: man:ntpd(8) Main PID: 799 (ntpd) CGroup: /system.slice/ntp.service └─799 /usr/sbin/ntpd -n -u ntp:ntp -g

Mar 01 15:02:26 oob-mgmt-server ntpd[799]: 1 Mar 15:02:26 ntpd[799]: Listen and drop on 1 v4wildcard 0.0.0.0:123 Mar 01 15:02:26 oob-mgmt-server ntpd[799]: 1 Mar 15:02:26 ntpd[799]: Listen normally on 2 lo 127.0.0.1:123 Mar 01 15:02:26 oob-mgmt-server ntpd[799]: Listen normally on 3 eth0 10.255.0.1:123 Mar 01 15:02:26 oob-mgmt-server ntpd[799]: Listen normally on 4 lo [::1]:123 Mar 01 15:02:26 oob-mgmt-server ntpd[799]: Listen normally on 5 eth0 [fe80::4638:39ff: fe00:100%2]:123 Mar 01 15:02:26 oob-mgmt-server ntpd[799]: Listening on routing socket on fd #22 for interface updates Mar 01 15:02:26 oob-mgmt-server ntpd[799]: 1 Mar 15:02:26 ntpd[799]: Listen normally on 3 eth0 10.255.0.1:123 Mar 01 15:02:26 oob-mgmt-server ntpd[799]: 1 Mar 15:02:26 ntpd[799]: Listen normally on 4 lo [::1]:123 Mar 01 15:02:26 oob-mgmt-server ntpd[799]: 1 Mar 15:02:26 ntpd[799]: Listen normally

on 5 eth0 [fe80::4638:39ff:fe00:100%2]:123 Mar 01 15:02:26 oob-mgmt-server ntpd[799]: 1 Mar 15:02:26 ntpd[799]: Listening on routing socket on fd #22 for interface updates

cu mulus@leaf01:m g mt-vrf:~$ netq show services mstpd

Matching services records: Hostname Service PID VRF Enabled Active Monitored Status Uptime Last Changed ------leaf01 mstpd 452 default yes yes yes ok 5h:10m:3s 4h:10m:58s leaf02 mstpd 454 default yes yes yes ok 5h:9m:41s 4h:10m:58s leaf03 mstpd 456 default yes yes yes ok 5h:10m:14s 4h:10m:58s leaf04 mstpd 467 default yes yes yes ok 5h:9m:54s 4h:10m:59s spine01 mstpd 447 default yes yes yes ok 5h:9m:39s 4h:10m:58s spine02 mstpd 454 default yes yes yes ok 5h:10m:17s 4h:10m:58s

System environmental validation Display & validate temperature

cumulus@oob-mgmt-server:~$ netq spine01 show sensors temp

Matching sensors records: Hostname Name Description State Temp Max Min Msg LastChngd ------spine01 psu1temp1 psu1 temp sensor ok 25 85 80 5 24m:19.467s spine01 psu2temp1 psu2 temp sensor ok 25 85 80 5 24m:19.467s spine01 temp1 board sensor near cpu ok 25 85 80 5 24m:19.467s spine01 temp2 board sensor near virtual switch ok 25 85 80 5 24m:19.467s spine01 temp3 board sensor at front left corner ok 25 85 80 5 24m:19.467s spine01 temp4 board sensor at front right corner ok 25 85 80 5 24m:19.467s spine01 temp5 board sensor near fan ok 25 85 80 5 24m:19.467s

Display & validate fan speed

cumulus@oob-mgmt-server:~$ netq spine01 show sensors fan

Matching sensors records: Hostname Name Description State Speed Max Min Message Last Changed ------spine01 fan1 fan tray 1, fan 1 ok 2500 29000 2500 38m:10.585s spine01 fan2 fan tray 1, fan 2 ok 2500 29000 2500 38m:10.585s

spine01 fan3 fan tray 2, fan 1 ok 2500 29000 2500 38m:10.585s spine01 fan4 fan tray 2, fan 2 ok 2500 29000 2500 38m:10.585s spine01 fan5 fan tray 3, fan 1 ok 2500 29000 2500 38m:10.585s spine01 fan6 fan tray 3, fan 2 ok 2500 29000 2500 38m:10.585s spine01 psu1fan1 psu1 fan ok 2500 29000 2500 38m:10.585s spine01 psu2fan1 psu2 fan ok 2500 29000 2500 38m:10.585s

Display & validate Power Supply (PSU)

cumulus@oob-mgmt-server:~$ netq show sensors psu

Matching sensors records: Hostname Name State Message Last Changed ------leaf01 psu1 ok 47m:42.330s leaf01 psu2 ok 47m:42.331s leaf02 psu1 ok 47m:50.247s leaf02 psu2 ok 47m:50.247s leaf03 psu1 ok 47m:45.496s leaf03 psu2 ok 47m:45.496s leaf04 psu1 ok 47m:37.158s leaf04 psu2 ok 47m:37.158s spine01 psu1 ok 47m:40.435s spine01 psu2 ok 47m:40.435s spine02 psu1 ok 47m:49.296s spine02 psu2 ok 47m:49.296s

Automation

Identify potential automation templates Tasks and information that are repeated often in multiple places or events are great candidates for templates and automation. Configuration information such as NTP, SNMP, and other settings may be common across every device in a network or at a given site. IP addresses, MAC addresses, interface configuration, and other items may exist with commonalities and minor differences that can be overcome. Commonly automated tasks:

·· Rapid Provisioning

·· Easy hardware swap for replacement

·· Configuration stored remotely and downloaded during provisioning

·· Elimination of manual cut and paste device configuration provisioning

FIGURE 49

·· Hot Swap Entire Device

·· Quick and easy device swapping, ZERO downtime

·· Automatically re-route traffic after incident and alert

·· Power off and remove device, install, connect and power-on new device

·· Provision automation applies appropriate configuration from the previous device

·· Easy, cost-effective spare and support strategy

FIGURE 50

·· Configuration Management

·· Configuring/etc/network/interfaces

·· Configuring daily messages (MOTD)

·· Configuring FRR

·· Interface and IP address templates

FIGURE 51

·· Application deployment and Programmatic changes

·· Proactive and reactive to network failures.

·· Automatic threat response

·· Action taken on fan failure or high temperature

·· Changes to multiple switches at once

Describe the principles of automation Manual configuration has the greatestNEGATIVE impact on the ability to scale your network. Reduce OpEx by avoiding “artisanal networking”, which is handmade, time-consuming, and expensive. Converged administration means converged toolsets faster deployment and improved problem resolution to automate common and repetitive tasks.

Version Control makes it easier to push configurations to a test environment that mirrors your production network.

Editing configuration files by hand can be prone to error, and you should first test all changes in a simulated lab environment. Contrary to intent-based networking hype, automation platforms cannot read your mind to determine what you MEANT to achieve. There are many benefits of automation.

·· Single points of configuration control across multiple platforms

·· Infrastructure as code

·· Versionable, testable, and repeatable configurations

·· Rapid deployment

Describe a library/module A module is a referenced non-volatile resource that defines one or more functions or classes which you intend to reuse in different codes of your program.

A library is a collection of non-volatile resources (modules) providing related functionality used by programs including configuration data, documentation, help information, message templates, pre-written code, classes, values, or type specifications. It is also a collection of implementations of repetitive behavior. Writing high level programs can reference the library to handle system calls rather than writing the same or similar code each time.

Describe groupings Grouping devices organizes them by commonalities for the ease of automation. For example all leafs may be in a group called leafs or all spine devices could be in a group named spines, since the members of each of those respective groups have many commonalities.

Further, platforms usually allow for the nesting of groups or multiple inclusion so that a device can belong to the groups; switches, cumulus, california, and leaf for example.

A basic Ansible hosts file example showing nested grouping is below.

[switches:children] cumulus eos ios

[cu mulus] cumulus01.example.net

[eos] eos01.example.net

[ios] ios01.example.net

Describe push vs. pull & agent vs. agentless Push vs. pull

In the automation push model, a user or process on a server sends commands to the local system for immediate execution. The push model depends on the clients being reachable during execution and the limitations are generated from the master.

In the automation pull model, an agent or the local system polls a server for information for non-immediate execution. In the pull model clients call the master when they are able to, providing increased scalability, but less control over when clients poll or receive information.

Agent vs. agentless

Agentless management requires no additional software and relies on existing process, such as SSH, to communicate with the control server. Ansible is an example of an agentless automation model.

Agent-based management requires installation of an application, or daemon, to communicate with the control server, but it also may use existing process such as SSH. NetQ has a python-based agent to install on monitored devices.

Describe idempotentcy Running the event multiple times will not change the outcome. The result of a successful performed event is independent of the numbers of times it is executed.

If a configuration is re-applied over and over the end result is the same configuration. The method of application may be impactful.

Name major automation vendors Ansible

Ansible is an open source, lightweight configuration management tool that can be used to automate many configuration tasks. Ansible does not require an agent be run on a switch; instead Ansible manages

nodes over SSH. Using Ansible, you can run automation tasks across many end points, whereas you use Mako within the context of a single switch. A particular script that runs a variety of tasks is referred to as a playbook in Ansible.

Puppet

Puppet is an open source tool that can automate configuration management through the use of a controller that syncs with agents installed on each end point. While similar in functionality to Ansible, Puppet relies on agents installed on each switch being managed, SSL certificates, and DNS to properly function. Puppet utilizes TCP port 61613 for syncing between agents and a controller. In Puppet, a particular script that runs a variety of tasks is referred to as a manifest, similar to the idea of an Ansible playbook.

Chef

Chef is an agent-based open source tool using Ruby in order to accomplish configuration management in a client-server architecture. Chef uses resources to make a recipe and collects those into cookbooks to apply a desired state to the network as defined in the recipes. Agents poll the information from master servers that respond via SSH.

SaltStack

Salt was designed to enable low-latency and high-speed communication for data collection and remote execution in sysadmin environments. The platform is written in Python and uses the push model for executing commands via SSH protocol. Salt allows parallel execution of multiple commands encrypted via AES and offers both vertical and horizontal scaling. A single master can manage multiple masters, and the peer interface allows users to control multiple agents (minions) directly from an agent. In addition to the usual queries from minions, downstream events can also trigger actions from the master. The platform supports both master-agent and de-centralized, non-master models.

CFEngine

The oldest and establish tool in configuration management written in C language. Lightweight agent system. Manages configuration of a large number of computers using the client–server paradigm or stand-alone. Any client state which is different from the policy description is reverted to the desired state. Configuration state is specified via a declarative language. CFEngine’s paradigm is convergent “computer immunology.”

Automation vendor table summary

Tool Agent Pull Push Configuration File Name App Store

Ansible No No Yes Playbook Galaxy

CFEngine Yes Yes No Policy Community Code Repository

Chef Yes Yes No Cookbook Supermarket

Puppet Labs Yes Yes No Manifest Forge

SaltStack Yes Yes Yes sls (salt state) Formulas

Articulate Linux automation strategy (push file restart –> service) Since Linux is maintained and configured through text files, those files can be modified with the changes taking effect upon restarting the service. The configuration is replaced and provides for eas of automation compared to the procedural impact of direct configuration commands.

One file can be referenced and transferred to the device rather than hundreds of NCLU commands in a procedural fashion. This method also provides commonality to existing Linux servers and infrastructure.

Enable and use the NCLU API Enabling API

To enable, start, or stop the HTTP API service, run the following systemd commands:

cumulus@switch:~$ sudo systemctl enable restserver cumulus@switch:~$ sudo systemctl start restserver cumulus@switch:~$ sudo systemctl stop restserver

There are two configuration files associated with the HTTP API services. The first configuration file is used for non-chassis hardware; the second for chassis hardware. Generally, only the configuration file relevant to your hardware needs to be edited, as the associated services determine the appropriate configuration file to use at run time.

·· /etc/nginx/sites-available/nginx-restapi.conf

·· etc/nginx/sites-available/nginx-restapi-chassis.conf

The HTTP API services are configured to listen on port 8080 for chassis hardware by default. However, only HTTP traffic originating from internal link local management IPv6s will be allowed. To configure the services to also accept HTTP requests originating from external sources:

1. Open /etc/nginx/sites-available/nginx-restapi-chassis.conf in a text editor

2. Uncomment the server block lines near the end of the file

3. Change the port on the now uncommented listen line if the default value, 8080, is not the preferred port, and save the configuration file

4. Verify the configuration file is still valid, and if the configuration file is not valid, return to step 1; review any changes that were made, and correct the errors

cumulus@switch:~$ sudo nginx -c /etc/nginx/sites-available/nginx- restapi-chassis.conf -t

5. Restart the daemons:

cumulus@switch:~$ sudo systemctl restart restserver

The IP:port combinations that services listen to can be modified by changing the parameters of the listen directive(s). By default, nginx-restapi.conf has only one listen parameter, whereas /etc/nginx/sites-available/ nginx-restapi-chassis.conf has two independently configurable server blocks, each with a listen directive. One server block is for external traffic, and the other for internal traffic. All URLs must use HTTPS, rather than HTTP. Do not set the same listening port for internal and external chassis traffic.

All traffic must be secured in transport using TLSv1.2 by default. Cumulus Linux contains a self-signed certificate and private key used server-side in this application so that it works out of the box, but Cumulus Networks recommends you use your own certificates and keys. Certificates must be in the PEM format.

NCLU API usage examples

To retrieve a list of all available HTTP endpoints.

cumulus@switch:~$ curl -X GET -k -u user:pw https://192.168.0.32:8080

To run net show counters on the host as a remote procedure call:

cumulus@switch:~$ curl -X POST -k -u user:pw -H “Content-Type: application/json” -d ‘{“cmd”: “show counters”}’ https://192.168.0.32:8080/nclu/v1/rpc

To add a bridge using ML2:

cumulus@switch:~$ curl -X PUT -k -u user:pw https://192.168.0.32:8080/ml2/v1/ bridge/”br1”/200

©2019 Cumulus Networks. All rights reserved. CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the “Marks”) are trademarks and service marks of Cumulus Networks, Inc. in the U.S. and other countries. You are not permitted to use the Marks without the prior written consent of Cumulus Networks. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis. All other marks are used under fair use or license from their respective owners. 09042019