BRKINI-2390.Pdf

BRKINI-2390

Data Center security within modern compute and attached fabrics - servers, IO, management

Dan Hanson Director UCS Architectures and Technical Marketing Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

How 1. Find this session in the Cisco Live Mobile App 2. Click “Join the Discussion” 3. Install Spark or go directly to the space 4. Enter messages/questions in the space

cs.co/ciscolivebot#BRKINI-2390

• BIOS and Kernel Manipulation

• Device Firmware Manipulation

• On Server Data Storage Manipulation

• Segmentation and Device access

• Root of Trust Flow

• Policy Control vs. Component Configuration (Cisco UCS and ACI)

• Example of Security Offloading: Skyport Systems

• Conclusion x86 Architecture Review Many Points of possible attack X86 Reference Legacy Elements

• You may see many terms in various articles shown here

• Over time, items moving on the CPU itself • Memory • PCIe

• Processors/Servers differentiation in some areas • Front Side Bus speeds • Direct Media Interface • Southbridge Configurations

• Not shown: Quick Path Interconnect/UltraPath Interconnect for CPU to CPU communications • Varied counts and speeds for multi-socket systems

• Current Designs have On-Die PCIe and Memory controllers • Varied numbers and DIMMs in memory channels by CPU

• Platform Controller Hub varies and can even offer server acceleration and security functions

BRKINI-2390 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8 Processing Cache Fundamental Caching on Processor • Representative diagram here • Caching was key to enhanced performance in industry • Caching in 3 levels typical on Intel and others • L1 is on-core and typically smallest with fastest access to CPU registers (scratchpad) • L2 is also on-core with larger size, but slower access than L1 • L3 shared over all cores on die • Data recursively searched from L1 through L3 (or L4) then DRAM

• This drawing is illustrative, showing Skylake layout example

• Memory and PCIe lanes now directly on CPU

• UPI for multi-socket shown here

• DMI3 for PCH connection

• Omni-Path HFI is on certain models for that IO technology

• Non-volatile basic storage for loading images

• Typically stored in dual bank flash (for updating) • BIOS rootkits are a method for attack

• Hardware initialization and boot loader

• Run time services to the Operating System

• Modern systems do not typically use BIOS after bootup • OS directly talk with hardware

• Cannot talk to external components typically

• Unified Extensible Firmware Interface – UEFI is a low level process run pre-boot (replaces the BIOS)

• UEFI spec 2.3.1 errata C (http://www.uefi.org/specs)

• UEFI can test images, including native OS loaders, prior to allowing them to execute

• Images are signed with OS Vendor Key (Lock-in FUD out there)

• UEFI can also define the trusted boot paths

UCS will then move onto next device in boot list if not trusted

• UEFI can halt the execution based on exceptions – or allow boot in non TXT mode

• UEFI allows for networking and storage functions at this low level

• UEFI drivers for our VIC Adapters coming – to support SAN booting

• Most modern CPUs support a hardware enforced protected mode

• This mode is only ring 0 in the diagram

• X86 CPUs support 3 rings with different privledges • Ring 0: OS kernel System Call • Ring 1,2: Device Drivers • Ring 3: User space

• Most OS will use only rings 0 and 3

• Transitions between rings at well defined calls only ** ** Recent press articles around ring 0 memory structures being directly accessible from ring 3 (circumvent system call to read data)

• Hypervisor offer shared ring 0 device access to Virtual Machines • Commonly called Ring -1

• Virtualization Extensions in CPU to assist hypervisor in handling memory page table switching, IO mapping, etc. System Call Ring -1 • Modern CPU VirtualizationExtensions also isolate data between these VMs • One VM cannot inspect other VM’s memory resident data • Hypervisor itself can (more on addressing these with newer extensions later) ** Recent press articles around ring 0 memory structures being directly accessible from ring 3 (circumvent system call to read data)

• Platform Controller has multiple capabilities to server system • Onboard LOM ports • Optional co-processing • USB access • PCI and PCIe access • Serial Peripheral Interface • Onboard storage like SATA/SAS • Low Pin Count bus for some items like management controller

• Attacks can be partially mitigated through BIOS control

• Key differentiation between desktop and server • Integrated IO to world • Ethernet • Async Serial • Storage (Flash, etc.) • Internal control of server • Environmentals and Power control • Storage controller config (on certain server models) • Virtual USB/Keyboard/Video/Mouse/Disk • Component firmware • Completely server vendor owned, and typically hardened

• Gen3 speeds shown • Gen3 roughly 2x of Gen2 • Gen2 roughly 2x of Gen1 • Went from 8B/10B to 128/130B encoding for less overhead in Gen3

• Multiple parallel lines with serial data

• Can have differing physical vs. electrical slots • Not all pins enabled example • x16 physically but x8 electrically

• Power specified along with speeds • 25W standard • 70W after poweron • 300 and 350W with added power pins

• Dedicated device for multi-disk configuration

• SATA/SAS links to disk, PCIe to host

• Caching of storage typical

• Original battery backup now typically supercap • Scale is to maintain data in cache over a weekend

• Advanced geometries of multiple units

• Caching of data in unencrypted format had typically led to targeting of this area

• Add-On Chip which Enables Secure Generation and Storage of Authentication Artifacts (Passwords/Certificates/Encryption Keys)

• Also provides pseudo-random values for various crypto functions

• Has Secure NVRAM location for storage of secret platform information (Checksums, Signed Keys, Platform Control Registers (PCRs), etc.)

• Installed with a one-way screw, and goal is to be tamper-resistant

• UCSM Inventory and Qualification Policies can trigger on TPM presence

• Establishes a Root of Trust for Measurement and Storage (RTM, RTS) on this NVRAM

• Advocates Suggest Geo-Location in Cloud to validate where services exist

• BIOS invokes boot loader for OS

• Traditional systems pointed to boot image on persistent media to execute

• System level OPROM and other utilities typically tied to physical peripheral in host

• Trusted boot added in modern system to measure and attest the validity of boot images

• Peripherals and Optimization Hardware with many options

• Inserts into the boot sequence

• Pre-boot environment to make low level configuration and operational validations

• Multiple human interactions typically with function keys to enter configuration utilities

• Security within this executing code on server left to peripheral vendor

• Firmware to be run on peripherals with processing capabilities

• Different but often matched to OS side specific drivers to kernel

• Increasing point of inspection and attack possibilities

• Open nature of systems leaves security enforcement model to peripheral vendor

• Software in OS that controls the hardware element

• Version maintained by OS and patched over time

• Typically plugs into OS side framework for configuration and control

• Maintained individually on machines often with manual process for driver maintenance

• Low level access provides attractive spot for attacks

• Both program code and data reference virtual memory and other elements

• This mapping of physical to virtual in multiple page tables is private to kernel

• Device driver access and memory structures also private

• Hypervisors abstract this further to virtualize these tables and data structures

• Very common attack vector to work in this area (gain access to private data)

• One commonality is speculation on code branch execution

• This speculation brings a tremendous performance enhancement for x86 devices • Processing resources can run results of multiple code-paths • Real code path decision can jump ahead to those results • Remnants of unused paths need to be cleaned up

• In these vulnerabilities, kernel mode speculation data within processor cache loose clarity on privilege state, and can leave artifacts which are inspected and measured within user space to learn about kernel execution and even data

• Three Vulnerabilties • Spectre Variant 1 (CVE-2017-5753) and Variant 2 (CVE-2017-5715) • Meltdown (CVE-2017-5754)

• Cisco PSIRT Advisory • https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa- 20180104-cpusidechannel • Medium severity partially because there are no known exploits • Impacts all Cisco UCS and Hyperflex servers • Closed systems may be vulnerable but don’t have an attack vector if there is no user code running on them (ex. UCS Fabric Interconnects)

• All three vulnerabilities require OS patching

• Spectre variant 2 (CVE-2017-5715) requires OS and processor microcode updates.

• Cisco UCS and Hyperflex Patch Points • UCS Manager 3.2(2), 3.1(3), 2.2(8) – Host Firmware impacted (B/C bundles) • Note that with UCS Manager 2.2(4) and later can have newer host firmware (B/C bundles) than infrastructure (A Bundle) firmware for M4 and M5 servers. • Cisco IMC Software 3.1(2) and 3.0(3) • M5 and M4 servers will post first with M3 and M2 servers to follow • Projected dates: Please see the Cisco PSIRT advisory

• Some operating systems also include microcode updates, although the microcode update wouldn’t be persistent across reboots (i.e. needs to load on each boot)

• Intel Update (Jan 11) – Patch for Broadwell and Haswell-based platforms may cause reboots - https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/

• Cisco testing is consistent with other industry reports. • Minimal impact on CPU or memory constrained workloads • Noticeable/Significant impact on IO constrained workloads such as databases • Biggest impacts on sequential read and sequential write workloads, especially with 4k block size.

• Industry Guidance • RedHat - https://access.redhat.com/articles/3307751 • Microsoft - https://cloudblogs.microsoft.com/microsoftsecure/2018/01/09/understanding-the- performance-impact-of-spectre-and-meltdown-mitigations-on-windows-systems/

• Cisco PSIRT - https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180104-cpusidechannel

• Intel - https://newsroom.intel.com/news-releases/intel-issues-updates-protect-systems-security-exploits/

• Google - https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html

• RedHat • https://access.redhat.com/security/vulnerabilities/speculativeexecution • https://access.redhat.com/articles/3307751

• Microsoft • https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/ADV180002 • https://cloudblogs.microsoft.com/microsoftsecure/2018/01/09/understanding-the-performance-impact-of-spectre-and- meltdown-mitigations-on-windows-systems/

• SuSE - https://www.suse.com/c/suse-addresses-meltdown-spectre-vulnerabilities/

• VMware • https://www.vmware.com/us/security/advisories/VMSA-2018-0002.html • https://kb.vmware.com/s/article/52085

• This diagram shows one well known custom firmware

• DD-WRT wrote open device firmware for D-Link, Linksys, and many other devices

• Enhancements to operation in open community • Support agreements typically negated once done

• This open and low level access an example in open system

• Injection at the boot image point to new code

• Completely moved to open security exposure

• Firmware updates in open systems may be signed by update utilities

• Utilities designed with desktop in mind • Challenge to scale to 100’s-1000’s of servers

• Traditionally methods over many servers did not look at how OS may inject custom code into firmware via open access points

• One method in industry is to measure these images constantly

BRKINI-2390 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34 On Server Data Storage Manipulation Processor Cache Manipulation Data and Code Inspection and Manipulation L1/L2/L3 Cache

• Cache data can exist simultaneously at these caches

• Advanced algorithms to populate caches in hopes of never having CPU find instructions/data an performance hit

• Methods to trick kernel segments to delay the cleanup are exploited

• Kernel system calls to transition from applications into dedicated kernel code and data access

• Hypervisor keep memory isolated between VMs

• Intel SGX to keep hypervisor from inspecting client memory data

• Future technologies on die to allow for encrypting and key management of memory pages

• Malicious code has been used to inspect memory in hosts directly (i.e. POS terminals)

• Data at rest on host desire to be secured

• Added resources to handle encrypt/decrypt

• Key management and preservation on server instance failure • Cannot loose data on device failure • Key persistence outside device

• Move up level to multi-drive encryption

• Key management across multiple devices

• Lower cost drives possible as function moved to controller

• UCS External Network External Communications Proxied Through Interfaces UCSM

• External Access to UCSM Mgmt Components Authenticated through PAM UCSM • UCS Internal Communications via Fabric Private Internal 127. Network Interconnect

• Component Layering – Unified Computing System Hardware CMC CIMC Fabric Blade Extender

BRKINI-2390 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41 UCS Control Plane Security TCP and UDP Ports to and from the UCS System . Inbound . CLI – Telnet (tcp/23) , SSH (tcp22) . HTML/XML – HTTP (tcp/80) , HTTPS (tcp/443) . CIM XML (tcp/5988) . SNMP – (udp/161) . IPMI over LAN – RMCP+ (udp/623), SOL – SSH (tcp22) . KVM – Avocent Video Session Protocol (tcp/2068),xmlPolicy (tcp/843) . Outbound . AAA – RADIUS (udp/1812 &1813), TACACS (tcp/49), LDAP (tcp/389) . TimeSync – NTP (udp/123) . SNMP Traps – SNMP(udp/162) . Call Home – smtp (tcp/25) . External Syslog – syslog (udp/514) . Name Resolution – dns (udp/53) . File Transfer – TFTP (udp/69), SFTP (tcp/115), FTP (tcp/20-21), SCP (tcp/21)

• Choose cipher strength High Medium Low Custom • Fully customizable • Choose TLS version(s) allowed • UCS 3.1.1 and later have CSDL componentry

• Map VLANs in UCS to uplinks / uplink sets

• Individual uplinks or group of uplinks or VPC

• Define a native VLAN per uplink grouping

• Can group a set of VLAN with common attributes

• Can then add the group to vNICs and templates

• Indirect control of VLAN optimization sets

• VLAN segmentation which provides L2 isolation between devices that are sitting within the same VLAN/subnet • Conserve VLAN and IP

• PVLAN construct are maintained between, physical, virtual device, and between UCS domain and external network • Primary • Community • Isolated • Promiscuous (Appliance Port only)

Promiscuous Router

Appliance Port Trunk Link (promiscuous port) Regular and PVLAN

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

UCS B200 M3

UCS 5108

SLOT SLOT 1 2

! Console Reset

SLOT SLOT 3 4

SLOT SLOT 5 6

SLOT SLOT 7 8

OK FAIL OK FAIL OK FAIL OK FAIL

VLAN List Uplink Group Server Interface UCS List Forwarding Boundary 10,11,12… E1/1,E1/2 Blade1 NIC0 & NIC1, Blade 2 NIC0

20,21,22… E1/10,E1/11 None

30,31,32… E1/20,E1/21 Blade 2 NIC1, Blade 3 NIC0 & NIC1

501 FC0,FC1 Blade 1 vHBA0, Blade 2 vHBA0

• Administratively defined by Security, Storage, and Network teams before use

• Template of full I/O rules (numbers of NICs and where they connect, numbers of HBAs and what they can see, etc.) valid for an organization

• Organization server administrator can attach the I/O sub-assembly to a server in a server definition • CANNOT however add a random adapter with other network/storage behind it, nor modify assembly

• L2/L3 Aggregation is needed above UCS

• L2 flows today only within UCS • No L3 switching possible within UCS

• L2 VLAN access tightly controlled in UCS • Administrator can set or block an untagged VLAN • Administrator can set list or block any VLAN tagging to restricted set • Administrator sets this per adapter or adapter template • These are enforced at the adapter hardware facing the OS or hypervisor • Raw socket writes will only be upper layer protocols – cannot manipulate hardware pinning tags as they are inserted and stripped on all hardware interfaces to/from UCS fabric • VLAN hopping attempts are dropped at the UCS adapter

• FCoE traffic can be assigned to its own VLAN • Can stop administrators from accidentally or maliciously creating adapters for monitoring

• Connection of a discreet server on UCS Fabric Interconnect will only work on “Appliance Port” • DCBXP negotiates whether to allow port to come online • UCSM configures what segments (and if native or not) are allowed on port • UCSM can optionally configure the MAC allowed on that port • System adds/removes any VNtags at this port as this is external

• Connection of a discreet Ethernet switch on the UCS fabric interconnect will only work on “Appliance Port” • Not supported in regular configurations • BPDU Reception will err-disable the port • When attacker attempts, they will not see any VNtags, nor any traffic for which they are not assigned • Appliance port is a VNtag end point itself

• UCS offers PVLAN functionality for bare-metal and VM-FEX deployments

• MAC-Forging can be disabled • Only administrator assigned MAC’s can be accepted

BRKINI-2390 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 53 Root of Trust Flow Baseline: Validation of Hardware as Root of Trust Cisco Anti-Counterfeit Technology (ACT and ACT2)

• Holds a Cisco Secure Unique Device Identifier (SUDI) certificate installed at manufacture

• Cisco IMC (Baseboard Mgmt Controller) measures it

• On success system allows access and start boot or secure boot process

• On failure is multipart process • Controlled disabling of management interfaces (to only show errored state and code to user) • Telnet/SSH, XML API, Web, KVM • Block server from booting in BIOS

• CHIMP installed in flash and SPI bootflash sectors are locked to read-only at manufacture

• CHIMP cannot be field upgraded

• CIMC boot certifies image is certified by Cisco • SHA512 encrypted hash of image • Included public keys for image decryption in locked flash

• Will then certify the supplemental flash (support apps)

• On new firmware install, certify new image prior to installation

• Customers cannot directly install firmware components

• Same process and Keys as for BIOS images

BRKINI-2390 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 56 UCS Server Hardware Anchor Image rd Signed Cisco and 3 Party Signing Software, Boot Loader Secure Run-time Integrity Validation Run-time Development Integrity

Secure Processor, Memory & Boot Secure ROM Boot Hardware Secure Storage Anchored Trust Entropy source with true ACT 2 randomization Lite Immutable Identity

ACT 2 + Secure Boot = Hardware Anchored Trust

Step 1 Step 2 Step 3 Step 4 Step 5

Hardware CPU CPU CPU CPU Anchor Boot0 Bootloader Application OS CHIMP Code (Boot1) Images Public Keys Stored Inside

Boot0 Boot0 Bootloader OS Launched Application Code checks Boot1 checks OS Images

Immutable Integrity Checks Image Signing

Built-in checks, CHIMP, Read only Flash Sector, Run time defense

BRKINI-2390 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 58 Policy Control vs. Component Configuration (Cisco UCS and ACI) BIOS & UEFI Firmware and Configuration Maintaining BIOS and UEFI, Need for Atomic Configuration