NSX-T Meets FRRouting – Part 1 Link Part 1 - https://rutgerblom.com/2020/01/17/nsx-t-meets-frrouting-part-1/ Part 2 - https://rutgerblom.com/2020/01/20/nsx-t-meets-frrouting-part-2/amp/ NSX-T Meets FRRouting – Part 1 rutgerblom January 17, 2020 2 Until recently I always used pfSense with the OpenBGPD package as the NSX- T Edge counterpart in my lab environment. It’s quick and easy to set up and works well enough. But pfSense is not what I typically find in a customer’s production environment. I started to investigate other virtualized “top-of-rack solutions” for the lab that would be a bit more similar to what I see in the enterprise. Right now I’m testing out FRRouting and I must say that I’m pretty impressed with this solution so far. At least it’s good enough to be the subject of a blog post or two � I’m going to walk through deploying and configuring a pair of FRRouting instances, the NSX-T Edge, and BGP routing in a lab environment. Follow along if you want. Target topology The diagram below shows a logical L3 design for the NSX-T Edge – FRRouting solution that we’ll be building: There’s nothing much out of the ordinary here. We have a Tier-0 gateway backed by two Edge nodes, and BGP routing. At the top of the diagram things look a bit less familiar with two Linux routers powered by FRRouting. That’s a nice sketch. Now let’s see if we can make it work too. Bill of materials The following software is used to build this environment: o NSX-T 2.5.1 o vSphere 6.7 U3 o Debian Linux 10.2 o FRRouting 7.2 Deploy FRRouting This first part is about getting the FRR instances up and running which begins with installing two Linux servers. Let’s get right to it. Install Linux servers Debian Linux is a good fit here as there is an official FRR Debian repository which makes installing FRR a lot easier. Each server is configured with two NICs. The ens192 interface is configured as the primary interface and will be the “north-facing” port. The ens224 interface is the “SDDC-facing” port. At this point we only assign a static IP address to the ens192 interface. The only additional components we need to install are the SSH server and standard system utilities: Complete the Debian installation on both servers. Install VLAN support The servers will soon be configured with some VLAN interfaces. To add support for this we install the VLAN package: apt install vlan -y Add the following line to /etc/modules so that VLAN (802.1Q) support is loaded during boot: 8021q Enable IPv4 packet forwarding We want the Linux servers to become Linux routers and as a part of that we need to enable IPv4 packet forwarding in /etc/sysctl.conf: net.ipv4.ip_forward=1 Reboot the servers after making this change. Configure network interfaces Time to configure the network interfaces on the Linux routers. The following shows the interface configuration per Linux router: frr-01: Interface IP address Comment ens192 10.2.129.101/24 Primary interface, north-facing ens224 – Secondary interface, SDDC-facing ens224.1611 172.16.11.253/24 Management VLAN ens224.1657 172.16.57.1/29 BGP peering VLAN ens224.1659 172.16.59.253/24 Overlay transport VLAN Which results in the following /etc/network/interfaces for frr-01: source /etc/network/interfaces.d/* # The loopback network interface auto lo iface lo inet loopback # The primary network interface - north-facing port auto ens192 allow-hotplug ens192 iface ens192 inet static address 10.2.129.101/24 gateway 10.2.129.1 dns-nameservers 10.2.129.10 dns-search demo.local # The secondary network interface - SDDC-facing port auto ens224 allow-hotplug ens224 iface ens224 inet manual mtu 9000 # The VLAN 1611 interface - Management auto ens224.1611 iface ens224.1611 inet static address 172.16.11.253/24 # The VLAN 1657 interface - BGP peering auto ens224.1657 iface ens224.1657 inet static address 172.16.57.1/29 # The VLAN 1659 interface - Overlay transport auto ens224.1659 iface ens224.1659 inet static address 172.16.59.253/24 frr-02: Interface IP address Comment ens192 10.2.129.102/24 Primary interface, north-facing ens224 – Secondary interface, SDDC-facing ens224.1611 172.16.11.254/24 Management VLAN ens224.1658 172.16.58.1/29 BGP peering VLAN ens224.1659 172.16.59.254/24 Overlay transport VLAN The corresponding /etc/network/interfaces for frr-02: source /etc/network/interfaces.d/* # The loopback network interface auto lo iface lo inet loopback # The primary network interface - north-facing auto ens192 allow-hotplug ens192 iface ens192 inet static address 10.2.129.102/24 gateway 10.2.129.1 dns-nameservers 10.2.129.10 dns-search demo.local # The secondary network interface - SDDC-facing auto ens224 allow-hotplug ens224 iface ens224 inet manual mtu 9000 # The VLAN 1611 interface - Management auto ens224.1611 iface ens224.1611 inet static address 172.16.11.254/24 # The VLAN 1658 interface - BGP peering auto ens224.1658 iface ens224.1658 inet static address 172.16.58.1/29 # The VLAN 1659 interface - Overlay transport auto ens224.1659 iface ens224.1659 inet static address 172.16.59.254/24 Restart the network to activate the new network interface configuration: systemctl restart networking Run the ip address command to verify that the new interface configuration is active: Install VRRP As you noticed we are “stretching” the management VLAN (1611) and the overlay transport VLAN (1659) between the Linux routers. Both routers can act as the default gateway for these VLANs at any given time. To make use of this capability we’ll set up VRRP with Keepalived. Install the package: apt install keepalived -y Create the Keepalived configuration file: /etc/keepalived/keepalived.conf. Below the Keepalived configuration per server: frr-01 (VRRP master): global_defs { # Email Alert Configuration notification_email { # Email To Address [email protected] } # Email From Address notification_email_from [email protected] # SMTP Server Address / IP smtp_server 127.0.0.1 # SMTP Timeout Configuration smtp_connect_timeout 60 router_id frr-01 } vrrp_sync_group VG1 { group { 1611 1659 } } vrrp_instance 1611 { # State = Master or Backup state MASTER # Interface ID for VRRP to run on interface ens224.1611 # VRRP Router ID virtual_router_id 10 # Highest Priority Wins priority 250 # VRRP Advert Intaval 1 Second advert_int 1 # Basic Inter Router VRRP Authentication authentication { auth_type PASS auth_pass VMware1!VMware1! } # VRRP Virtual IP Address Config virtual_ipaddress { 172.16.11.1/24 dev ens224.1611 } } vrrp_instance 1659 { # State = Master or Backup state MASTER # Interface ID for VRRP to run on interface ens224.1659 # VRRP Router ID virtual_router_id 11 # Highest Priority Wins priority 250 # VRRP Advert Intaval 1 Second advert_int 1 # Basic Inter Router VRRP Authentication authentication { auth_type PASS auth_pass VMware1!VMware1! } # VRRP Virtual IP Address Config virtual_ipaddress { 172.16.59.1/24 dev ens224.1659 } } frr-02 (VRRP backup): global_defs { # Email Alert Configuration notification_email { # Email To Address [email protected] } # Email From Address notification_email_from [email protected] # SMTP Server Address / IP smtp_server 127.0.0.1 # SMTP Timeout Configuration smtp_connect_timeout 60 router_id frr-02 } vrrp_sync_group VG1 { group { 1611 1659 } } vrrp_instance 1611 { # State = Master or Backup state BACKUP # Interface ID for VRRP to run on interface ens224.1611 # VRRP Router ID virtual_router_id 10 # Highest Priority Wins priority 150 # VRRP Advert Intaval 1 Second advert_int 1 # Basic Inter Router VRRP Authentication authentication { auth_type PASS auth_pass VMware1!VMware1! } # VRRP Virtual IP Address Config virtual_ipaddress { 172.16.11.1/24 dev ens224.1611 } } vrrp_instance 1659 { # State = Master or Backup state BACKUP # Interface ID for VRRP to run on interface ens224.1659 # VRRP Router ID virtual_router_id 11 # Highest Priority Wins priority 150 # VRRP Advert Intaval 1 Second advert_int 1 # Basic Inter Router VRRP Authentication authentication { auth_type PASS auth_pass VMware1!VMware1! } # VRRP Virtual IP Address Config virtual_ipaddress { 172.16.59.1/24 dev ens224.1659 } } Restart the Keepalived service on both routers to activate the new configuration: systemctl restart keepalived We can now verify VRRP operation by running systemctl status keepalived: Running the ip address command will hopefully show the virtual IP address on the two VLAN interfaces: And a ping to the virtual IP address from the VRRP backup node (frr-02 in this case) should be successful: Install FRRouting With Linux installed and configured we continue with the FRRouting installation. Begin by adding the FRR Debian repository: curl -s https://deb.frrouting.org/frr/keys.asc | apt-key add - FRRVER="frr-stable" echo deb https://deb.frrouting.org/frr $(lsb_release -s -c) $FRRVER | tee -a /etc/apt/sources.list.d/frr.list apt update && apt install frr frr-pythontools -y FRRouting is now installed. Configure FRRouting capabilities We only enable the routing protocols that are needed. To make FRR a good match for the NSX-T Edge we would like the instances to be capable of doing BGP and BFD. So we simply enable these daemons in /etc/frr/daemons. bgpd=yes bfdd=yes Restart the FRR service and verify that the BGP and BFD daemons are active: systemctl restart frr systemctl status frr This is looking good. The FRR instances are now ready for control plane configuration. Summary This completes part 1 of the series on NSX-T and FRRouting. We’ve been quite productive: o Installed two Debian Linux servers o Installed VLAN support o Enabled packet forwarding o Configured network interfaces o Installed and configured VRRP o Installed FRRouting In the next part we’ll continue with deploying the NSX-T Edge and setting up BGP routing between NSX-T and the FRR instances.