Give Your PXE Wings! It’s not magic! How actually works.

Presentation for virtual SREcon 2020 By Rob Hirschfeld, RackN Rob Hirschfeld

@zehicle Co-Founder of RackN

We created Digital Rebar Bare Metal Provisioning ++

@L8istSh9y Podcast on PXE: http://bit.ly/pxewings In concept, O/SO/S Provisioning is Easy!

We’re just installing an on a server or switch!

Why is that so hard?!

● Bootstrapping ● Limitations ● Variation ● Networking ● Security ● Performance Computer ● Post configuration In concept, Pre- & Provisioning is Easy! Post- Config We’re just installing an operating system on a server or switch!

Why is that so hard?!

● Bootstrapping ● Firmware Limitations And that’s not even including: ● Variation ● Networking ● System Inventory ● Security ● System Validation ● Performance ● Hardware Configuration ● Post configuration ● Naming & Addressing ● Credentials Injection Exploring Provisioning Approaches

Netboot (25 min) Image Deploy (10 min) Esoteric Flavors (5 min)

● PXE ● Packer ● kexec ● iPXE ● Write Boot Part ● Secure Boot ● ONIE ● Cloud Init ● BMC Boot ● Kickstart ● Preseed Exploring Provisioning Approaches

Netboot (25 min) Image Deploy (10 min) Esoteric Flavors (5 min)

● PXE ● Packer ● kexec ● iPXE ● Write Boot Part ● Secure Boot ● ONIE ● Cloud Init ● BMC Boot ● Kickstart ● Preseed

All roads lead to a kernel init process PXE PXE Let’s PXE! Bootstrapping is a multi-stage process

Server DHCP Firmware PXE NextServer & Options TFTP Bootloader Stage 1 lpxelinux.0 Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi

HTTP(S) O/S Kernel O/S ISO First: Get on the network

Server DHCP Firmware PXE NextServer & Options

Provisioning Service(s) Then: Download a Bootloader

Server

TFTP Bootloader Stage 1 lpxelinux.0 Provisioning Service(s) Then get a BETTER Bootloader

Server

Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi Finally load a “real” operating system

Server

Provisioning Service(s)

HTTP(S) O/S Kernel O/S ISO Each stage is actually a NEW O/S Load

Server DHCP Firmware DHCP PXE NextServer & Options TFTP Bootloader DHCP Stage 1 lpxelinux.0 Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi DHCP

HTTP(S) DHCP O/S Kernel O/S ISO And modern servers can skip TFTP! So… technically, no longer PXE

Server DHCP Firmware PXE NextServer & Options

Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi

HTTP(S) O/S Kernel O/S ISO Yay! We’re done, right? Provisioning is more than PXE

Server iPXE O/S Kernel Bootloader O/S ISO

Config Kickstart Templates Provisioning Service(s) Download Installation Packages

Access & Post-Config Apps Hardware varies, so Install must be guided by templates

Server

O/S Kernel

Config Kickstart Templates Provisioning Service(s) ISOs are minimal and stale So they must be updated

Server

O/S Kernel

Kickstart Provisioning Service(s) Download Repo Installation Packages Mirrors And then you can actually connect to start configuring the system!

Server

O/S Kernel

Kickstart Provisioning Service(s)

Installation

Access & Post-Config Apps

Automating Provisioning means Connecting all these steps together

Server iPXE O/S Kernel Bootloader O/S ISO

Config Kickstart Templates Provisioning Service(s) Download Installation Packages

Access & Post-Config Apps

But wait…. There’s more to consider!

Server iPXE Out of Band Management (BMC, IPMI, Redfish, etc) O/S Kernel Bootloader O/S ISO Infrastrastructure Config Kickstart as Code Templates Provisioning Service(s) Download Installation Packages

Access & Post-Config Apps IaC? Show us some templates! Typical PXE Questions

Why is this so fragile?

What about PXE over Wifi?

What about using a VLAN?

Can I dockerize this?

What about setting BIOS & RAID?

How can I make this faster? How can we Server PXE/iPXE PXE simplify that?!! Small RAM only Footprint Provisioning O/S Inventory At RackN, we’ve been using an in Service(s) memory operating system, “sledgehammer,” based on CentOS. Informed Guided Installation Installation It’s highly optimized to

● Run on nearly any hardware ● Load very quickly ● Collect deep inventory ● Have built-in tools for system tasks like hardware config Image Based Deployment (10x faster!)

Server PXE/iPXE PXE

RAM only Small O/S Footprint Provisioning Write O/S O/S Image Service(s) To Drive(s) as Archive

reboot Informed Machine Installation Init Immutable Provisioning

Server

PXE PXE/iPXE Highly RAM only Minimal Available O/S Footprint Provisioning Service(s) Config & Machine Attach Disks Initialize

Load Apps Container & Containers Registries And now… Advanced Provisioning! ESXi Provisioning

Server iPXE O/S Kernel Bootloader O/S ISO Provisioning Config weasel Service(s) Templates

restricted Access & CLI/python Apps

Control via VMware ESXi VMw APIs Tooling ONIE: Open Network Install Environment

Switch

Current DHCP Firmware Provisioning Service(s) New HTTP(S) Firmware O/S Image

Designed for Embedded Systems where we’re replacing the O/S as a complete image.

Does have DHCP options for a startup script. kexec (kernel execute)

Server Normal Running O/S Provision Download New Kernel New Kernel kexec Starts Provisioning Without New O/S Service(s) Rebooting kexec (kernel execute)

Server Normal Running O/S Provision Download New Kernel New Kernel kexec Provisioning New O/S Service(s) Start kexec iPXE New Normal Installation Provision Without Rebooting Secure Boot Required SIGNED Bootloaders

Server DHCP Secured Firmware NextServer & Options Enabled TPM HTTP(S) Trusted Provisioning  Bootloader Signed ipxe.efi Service(s) Verified HTTP(S) Trusted O/S Kernel Signed O/S ISO Verified BMC Boot option 1

Server Install BMC Firmware Media

Attached O/S Install Provisioning Service(s) Kickstart Download Installation Packages Post-Config Access & Apps BMC Boot option 2

Server Install BMC Firmware Media

Bootloader DHCP Provisioning Config Service(s) Kickstart Templates Download Installation Packages Post-Config Access & Apps Thanks!

Contact us:

Rob Hirschfeld, RackN.com

Digital Rebar Behind the Firewall, Self-Service Infrastructure as Code

Self-Trials: rebar.digital