City of Arlington Council Agenda Bill

Item: CA #5 City of Arlington Attachment Council Agenda Bill D

COUNCIL MEETING DATE: July 6, 2021 SUBJECT: Request for additional funding for network equipment from outcome of network assessment. ATTACHMENTS: Network assessment results and proposal for network switching replacement. DEPARTMENT OF ORIGIN IT; Bryan Terry, Director 360-403-4610 EXPENDITURES REQUESTED: $30,346.87 BUDGET CATEGORY: Program Development BUDGETED AMOUNT: None – Additional costs above budgeted amount LEGAL REVIEW: DESCRIPTION: Council is asked to consider the replacement of network switching as an outcome of the network assessment that was performed earlier this year.

HISTORY: City Council approved a formal network assessment earlier this year. The outcome of that assessment was to replace some existing switching to avoid a bottleneck situation when deploying redundant server/storage at the Public Works building.

ALTERNATIVES:

RECOMMENDED MOTION: I move to approve the budget amendment for network equipment in the amount of $30,346.87.

City of Arlington

Network Assessment

June 8th, 2021

Prepared by: RSI Professional Services Table of Contents

Summary ...... 3 Overview ...... 3 Network Assessment ...... 3 Section 1: Hardware Overview & Recommendations ...... 3 End of Life Status ...... 3 Hardware & Topology ...... 4 Section 2: Configuration Review & Recommendations ...... 5 Contact Information ...... 7 Glossary ...... 7

CITY OF ARLINGTON NETWORK ASSESSMENT | RSI Professional Services

Summary City of Arlington has asked Right! Systems, Inc. (RSI) review the network configurations of their switching infrastructure and provide any recommendations based on current best practices. This assessment includes evaluation of switch hardware, logical design, and configuration.

Overview The City of Arlington network switching infrastructure focused on during this assessment consists of Aruba/HP switches of 3 general model types. The assessment performed by the RSI engineer encompassed the switches located within the City of Arlington network based on IP information provided by Adam (CoA). This assessment did not look at the Sonicwall firewall utilized within the network as it is actively apart of a firewall replacement project by another engineer.

Network Assessment RSI has broken this portion of the review into two (2) sections: Hardware Health & Current Status and Configuration Review & Recommendations against current best practices. For the logical design, configurations from all devices were taken to understand configuration details and network device relationships. Section 1: Hardware Overview & Recommendations A review of switching hardware was performed with the intent of determining the current health and potential risks to the City of Arlington network. Based on this review, the following comments and recommendations are provided: End of Life Status • J9148/J4146A models with PoE+ standards for the 2910 series were announced as End of Sale May 31,2014 and end of support May 31, 2019. These switches make up ~80% of the IDF layer switches. • J8692A/J8963A models with PoE+ standards for the 3500 series were announced as End of Sale June 30, 2014 and end of support June 30, 2019. These switches make up ~15% of the IDF layer switches. rd o While they are covered by a lifetime warranty through a 3 party, the devices owned by City of Arlington are not entitled to software/firmware updates from HPE and are in violation of the Software EULA and Hardware Warranty agreement (included in document package). The HPE Warranty clearly states the following: TRANSFER OF HEWLETT PACKARD ENTERPRISE NETWORKINGHARDWARE PRODUCTS TO ANY THIRD PARTY OTHER THAN THE ORIGINAL BONA FIDE END USER VOIDS THE HEWLETT PACKARD ENTERPRISE NETWORKINGPRODUCT WARRANTY TO THE FULLEST EXTENT ALLOWED BY LAW • RECOMMENDATIONS: o Discontinue purchasing “Gray Market” hardware that is not covered by manufacturer warranty

CITY OF ARLINGTON NETWORK ASSESSMENT | RSI Professional Services

o Invest in modern network hardware that includes active firmware development for patching of security vulnerabilities and software defects, as well as “official” hardware replacement and warranty services from the manufacturer Hardware & Topology • Core Site Nutanix Deployment and Remote Site Backups o As City of Arlington is pursuing best practices, bringing the current core switching up to recommended best practices and having a remote site backup to help protect critical data in event of a catastrophic failure is strongly recommended. • RECOMMENDATIONS: o Utilize similar hardware for your server and storage design. As Nutanix is already used in the City of Arlington environment, this would be a favorable option for a back up location for a hyper converged environment. o For switching hardware, the following recommendations are based off the following objectives; provide redundancy, maintain uptime, and avoid risks of catastrophic failure with the Nutanix hyper converge environment in the event of a single switch failure: . Recommended hardware build - 2x Aruba 6300M – 24 Port SFP+ & Aruba 6300M – 24 Port PoE+ Multi-Gigabit • Pair of Aruba 6300M SFP+ switches for core deployment to align to best practices o Recommended redundant power supplies, rack mount kit, and DAC cables for backplane capacity • A minimum of one (1) Aruba 6300M for auxiliary site to ensure ability to recover from critical failures and comply with best practices o Recommended redundant power supplies and rack mount kit • For these deployments, 4 SFP+ transceiver modules will be required for the Aruba switches; J9151E model LC SFPs are suggested. . This is the current generation of switching, running the latest CX OS version, and currently should provide the greatest longevity. • Device Modularity o The majority of switches currently deployed do not offer field-replaceable, redundant power supplies. Should the integrated power supply in the switch fail, the switch would remain offline until a replacement device could be procured and installed. Modern switches offer modular power supplies that are field-replaceable allowing the failed part to be serviced without having to completely replace the switch o The majority of the access-layer switches deployed today do not have dedicated stacking interfaces or backplane, meaning they consume standard interfaces on each switch to form a logical stack with less capacity. Newer switches offer dedicated stacking interfaces that allow for faster interconnect speeds between switch members in a stack, and do not consume standard interfaces on the switch to build the stack, allowing for higher port utilization for end devices and infrastructure connections as well as simpler management. • RECOMMENDATIONS:

CITY OF ARLINGTON NETWORK ASSESSMENT | RSI Professional Services

o When replacing failed or end of life hardware, purchase hardware that includes dedicated stacking interfaces, field-replaceable power supplies, and high-speed uplink interfaces • Uplink Performance o The majority of connections uplinking from each access-layer switch back to the network core are 1Gig uplinks currently. Most client devices today support speeds greater than 100mbps for both their wired and wireless performance. With these kinds of speeds coming from clients, uplink congestion can quickly become a limiting factor. . Exceptions are COAPWSW3930 (Public Works Admin Bldg) & COAWOSW3386 (Water Office) which have 10G uplinks connecting them to the core switch stack. • RECOMMENDATIONS: o AT A MINIMUM - redundant uplinks where possible are strongly encouraged to help with resiliency and further increased capacity with link aggregation. o Where possible, upgrade uplink speed from 1Gig to 10Gig from IDF switches back to the 5406 core switches, and ensure the connections are redundant between two core switches. Not only does this design allow for fault tolerance, but increases the overall traffic capacity, reducing or eliminating the potential uplink bottleneck. Section 2: Configuration Review & Recommendations • A number of best practices are already in place and configured correctly o LLDP, STP, VSF (for the core), and static routes are summarized as best as they can be at this time. Telnet is disabled, VLAN naming scheme is easy to understand, and the configurations are kept simple across switches in their design. • RECOMMENDATIONS: o LDAP integration for authentication to network switches . Allows for the creation of a service account to be used which can be managed by AD. Should a need to change the password access to those switches come up, one can simply change the credentials for a single user in AD instead of local credentials for each device. . Auditability of who did what, and when becomes much easier when leveraging AAA via LDAP, Radius or TACACS o Network segmentation between different site locations . A good practice for environments with multiple sites is to segment out the networks to avoid having single networks cover a larger geographical area. For example, assigning a /16 network to City Hall and breaking it down into separate /24 networks for each type of VLAN (management, voice, data, wireless, etc.). This process being repeated at each location allowing for more granular control and isolation within the network. . Transitioning to a distributed/segmented network that leverages Layer 3 (routing) to connect each remote facility adds a layer of protection against Spanning Tree events, where a simple misconfiguration or accidental cabling loop can bring down the whole network. When leveraging routed links between

CITY OF ARLINGTON NETWORK ASSESSMENT | RSI Professional Services

locations, any spanning tree event is isolated to the remote facility, and is unable to impact the rest of the network o General configuration cleanup and optimization . There are configuration elements that can be removed from devices as they are configured, but not in use. This adds the risk of future unexpected behaviors to occur during configuration changes. An example of this is on the core switch, 10.1.1.6, RIP is configured and enabled, but not actually peering with another device at this time. . Securing unused interfaces by putting them in a disabled/down state . Creating and applying a ‘black hole’ vlan to unused interfaces, . Update the interface descriptions to ‘UNUSED’ or ‘AVAILABLE’ o Network Access Control for wired and wireless users . Being able to prevent and control unauthorized access to network services is a critical component to overall network security. The previous recommendations are building blocks for Network Access Control (NAC). This technology controls access to the network based on information gathered from the device. If the device/user is a known employee, the user is granted more access to the network. If the user/device is unknown, then either the user is granted no access, or only access to the Internet.

CITY OF ARLINGTON NETWORK ASSESSMENT | RSI Professional Services

Contact Information Name Title Phone # Email Project Role Reilly Aiden Network 360.561.5900 [email protected] Network Engineer Assessor

Glossary Broadcast Domain A broadcast domain is a logical division of a computer network, in which all nodes can reach each other by broadcast at the data link layer. A broadcast domain can be within the same LAN segment or it can be bridged to other LAN segments. (see VLAN)

CIDR (notation) – Classless Inter-Domain Routing A method and syntax for defining network subnet sizes. CIDR allows for the division of classful supernets into smaller subnets. CIDR notation is typically expressed by using “/xx” where “xx” is the routing prefix. For example, the subnet mask 255.255.255.0 is expressed in CIDR notation by /24.

CLI – Command Line Interface A CLI is a blanket term used to describe any hardware/software interface that uses only text.

CoS – Class of Service Class of service is a parameter used in data and voice protocols to differentiate the types of payloads contained in the packet being transmitted. The objective of such differentiation is generally associated with assigning priorities to the data payload. CoS is a layer-2 concept. (see QoS)

IDF – Individual Distribution Frame A network room in a building that primarily provides access connections. Typically, an IDF will only make a remote connection to a BDF or MDF.

Interface, Routed A Routed Interface operates on layer-3 of the OSI model and is used to communicate between diverse IP subnets. By design, a routed interface cannot forward broadcast. In order to forward multicast traffic, a multicast routing protocol (like PIM) must be configured.

Interface, Switched A Switched Interface operates on layer-2 of the OSI model and is used to communicate between hosts on the same IP subnet(s). By design, a switched interface cannot route traffic between diverse IP subnets. It should be noted that on a network switch, a layer-3 interface could be assigned to a VLAN that is in turn assigned to physical ports. In this case, the physical ports are still switched interfaces, with a logical routed interface.

LACP – Link Aggregation Control Protocol LACP provides a method for LAG connections to dynamically add and remove physical link members without additional configuration or service interruption.

LAG – Link Aggregation Group A protocol standard that allows one or more equal-speed physical network links to act as a single logical

CITY OF ARLINGTON NETWORK ASSESSMENT | RSI Professional Services

link. When multiple physical links and a control protocol is used (see LACP), LAG provides a logical link with improved speed (the sum of the speeds of the physical links) and resiliency (the logical link will continue to function so long as a single physical link is functioning). It should be noted that while the overall speed is the sum of the LAG member links, that the bandwidth of any single session will still be limited to the speed of an individual link. That is to say that a 4Gb/s LAG formed from 4 1Gb/s links will still have a single session speed limit of 1Gb/s. So you get a bonded interface of (X)Gbe with LAG. But any single session will only ride a single link connection until transmission is completed between that host and destination. So no one host or session has the full total bandwidth of all the links in the LAG combined. “(X)” GB connection, only the bandwidth of a single link with in the LAG.

LAN – Local Area Network A network that interconnects multiple hosts in the same geographical area. Typically, a LAN will utilize private addresses (see “Private Address Space”) and not include service provider connections.

Layer-1 In reference to the network OSI model, layer-1 (physical layer) is where physical network connections are made. Wires, pins, voltage, modulation, etc…

Layer-2 In reference to the network OSI model, layer-2 (data link layer) is where two hosts that are connected via layer-1 establish a communication link. This layer is responsible for the interface between the host and the raw data transmitted over layer-1, as well as error checking and packet synchronization. VLAN’s define layer-2 segments, also known as broadcast domains.

Layer-3 In reference to the network OSI model, layer-3 (network layer) provides the communication means between two hosts on the same network. Layer-3 is where the IP address is defined.

NAC – Network Access Control NAC describes a suite of policy-based detection and remediation tools used to secure an enterprise network.

PoE – Power over Ethernet PoE is defined by the 802.3af and 802.3at IEEE protocol standards and defines how network devices deliver power over network cabling network clients. The amount of power delivered depends on the infrastructure device and can range from 15.4 watts to 25.5 watts.

QoS – Quality of Service In networking terms, QoS refers to the design and implementation of switch configuration that defines traffic prioritization metrics. Typically, QoS is used to identify and give preference to traffic that is susceptible to latency (the length of time it takes to reach a destination), packet loss (the loss of packets due to connection or performance issues), and jitter (the variance in latency). Susceptible traffic includes voice and video services.

Redundancy As a network concept, redundancy is a duplication of hardware meant to provide continuous service in the event of hardware failure. Depending on the hardware, the duplicate unit may be in an active or standby state.

CITY OF ARLINGTON NETWORK ASSESSMENT | RSI Professional Services

Resiliency As a network concept, resiliency is the ability of a network to dynamically heal in the event of a hardware failure, outage, or other network event. Resiliency is achieved through appropriate network design and the implementation of dynamic routing protocols.

Route Engineering Route engineering is the process of manipulating traffic paths to achieve specific goals. The process of route engineering involves identifying objectives, then creating rules and filters to direct network traffic in specific, predictable ways. This typically includes automatic and manual path selection methods. In many cases, route engineering is used to send specific traffic on a path that a dynamic protocol would typically not select.

SSH – Secure Shell A network protocol used for encrypted terminal access to various hosts.

STP – Spanning-Tree Protocol In modern networks, STP is a term used to describe any layer-2 loop prevention protocol based on spanning-tree concepts. Basic spanning-tree uses information packets called Bridge Protocol Data Units (BPDU’s) to identify and block network loops. Many variations of the spanning-tree protocol exist, including Multiple Spanning-Tree (MST), Per-VLAN Spanning-Tree (PVST and PVST+), and Rapid Spanning-Tree Protocol (RSTP).

Telnet A network protocol used for unencrypted terminal access.

VLAN – Virtual Local Area Network (See “802.1q”)

VoIP – Voice over IP This term refers to a bevy of technologies developed to facilitate bi-directional voice and multimedia communications over IP networks. This term is typically used to describe digital phone systems and the supporting software that operates them.

CITY OF ARLINGTON NETWORK ASSESSMENT | RSI Professional Services

Q U O T E Right! Systems, Inc. Date 06/14/21 Quote # RSIQ55716-04 11911 NE Ist Street SalesRep Suite 212 Carrie Sovde (206) 271-8866 Bellevue, WA 98005 Prepared By Shinji Carmichael (503) 972-2641 Phone: (206) 271-8866 Customer Contact Bryan Terry Notes (360) 403-4610 [email protected]

Customer Bill To Ship To City of Arlington Administration City of Arlington Administration City of Arlington Administration Bryan Terry (360) 403-4610 Bryan Terry (360) 403-4610 Bryan Terry (360) 403-4610 238 N. Olympic Avenue 238 N. Olympic Avenue 238 N. Olympic Avenue Arlington, WA 98223 Arlington, WA 98223 Arlington, WA 98223 United States United States United States Terms: Ship Via: NET 30 Ground Special Instructions: Description: NASPO Datacom Contract AR3228, WA Contract # 05819 HPE Switching - Nutanix Cluster Deployment

# Description Part # Qty Unit Price Ext. Price 1 Nutanix Cluster - HPE Aruba 6300M - 24 Port SFP+

2 HPE Aruba 6300M - Switch - L3 - managed - 24 x 1 Gigabit / 10 Gigabit JL658A 2 $8,527.53 $17,055.06 SFP+ + 4 x 1 Gigabit / 10 Gigabit / 25 Gigabit / 50 Gigabit SFP56 (uplink / stacking) - front and side to back - rack-mountable

3 HPE Aruba X371 - Power supply - hot-plug / redundant - AC 100-240 V - JL085A#B2B 4 $247.16 $988.64 250 Watt - United States - PDU Power Cord

4 Aruba X414 1U Universal 4-post Rack Mount Kit J9583B 2 $109.29 $218.58

5 HPE Aruba Direct Attach Cable - 25GBase direct attach cable - SFP28 to JL487A 2 $119.76 $239.52 SFP28 - 2 ft

6 HPE Aruba Central Device Management - Subscription license (5 years) - 1 JY927AAE 2 $154.63 $309.26 token - hosted - ESD

7 SubTotal $18,811.06

8 Intermediary Site - HPE Aruba 6300M - 24 Port PoE+ mGig

9 HPE Aruba 6300M - Switch - L3 - managed - 24 x 1/2.5/5/10GBase-T + 4 x JL660A 1 $6,726.15 $6,726.15 1 Gigabit / 10 Gigabit / 25 Gigabit / 50 Gigabit SFP56 (uplink / stacking) - front and side to back - rack-mountable - PoE+ (1440 W)

10 HPE Aruba X372 - Power supply - hot-plug / redundant (plug-in module) - JL086A#B2B 2 $375.74 $751.48 AC 100-240 V - 680 Watt - United States - PDU Power Cord

11 Aruba X414 1U Universal 4-post Rack Mount Kit J9583B 1 $109.29 $109.29

12 HPE Aruba Central Device Management - Subscription license (5 years) - 1 JY927AAE 1 $154.63 $154.63 token - hosted - ESD

13 SubTotal $7,741.55 Sub Total $26,552.61

Authorized Signature Title Date

By signing and dating the above referenced quote, customer authorizes purchase and agrees to Right! Systems terms and conditions.

Terms and Conditions Right! Systems Inc. Standard Terms and Conditions apply. Terms are N30 OAC. Applicable sales tax and freight are excluded and will be calculated at the time of shipping unless specifically requested. Pricing is valid until the end of each month and pricing may be subject to change. All returns are subject to authorization and will be subject to a 15% restocking fee. A copy of our standard Terms and Conditions may be requested by contacting 1-800-571-1717.

06/14/21 © 2015-2021 Right! Systems Inc. Page Jared Luther v03.032217 Opportunity #: 97930 Reference # N/A 2 of 2 Q U O T E Right! Systems, Inc. Date 06/08/21 Quote # RSIQ55980-01 11911 NE Ist Street SalesRep Suite 212 Carrie Sovde (206) 271-8866 Bellevue, WA 98005 Prepared By Shinji Carmichael (503) 972-2641 Phone: (206) 271-8866 Customer Contact Bryan Terry Notes (360) 403-4610 [email protected]

# Description Part # Qty Unit Price Ext. Price 1 Axiom J9151E-AX - SFP+ transceiver module (equivalent to: Aruba J9151E) - J9151E-AX 4 $309.39 $1,237.56 10 GigE - 10GBase-LR - LC single-mode - up to 6.2 miles - 1310 nm Sub Total $1,237.56

Authorized Signature Title Date

By signing and dating the above referenced quote, customer authorizes purchase and agrees to Right! Systems terms and conditions.