Give Your PXE Wings! It’s not magic! How booting actually works. Presentation for virtual SREcon 2020 By Rob Hirschfeld, RackN Rob Hirschfeld @zehicle Co-Founder of RackN We created Digital Rebar Bare Metal Provisioning ++ @L8istSh9y Podcast on PXE: http://bit.ly/pxewings In concept, O/SO/S Provisioning is Easy! We’re just installing an operating system on a server or switch! Why is that so hard?! ● Bootstrapping ● Firmware Limitations ● Variation ● Networking ● Security ● Performance Computer ● Post configuration In concept, Pre- & Provisioning is Easy! Post- Config We’re just installing an operating system on a server or switch! Why is that so hard?! ● Bootstrapping ● Firmware Limitations And that’s not even including: ● Variation ● Networking ● System Inventory ● Security ● System Validation ● Performance ● Hardware Configuration ● Post configuration ● Naming & Addressing ● Credentials Injection Exploring Provisioning Approaches Netboot (25 min) Image Deploy (10 min) Esoteric Flavors (5 min) ● PXE ● Packer ● kexec ● iPXE ● Write Boot Part ● Secure Boot ● ONIE ● Cloud Init ● BMC Boot ● Kickstart ● Preseed Exploring Provisioning Approaches Netboot (25 min) Image Deploy (10 min) Esoteric Flavors (5 min) ● PXE ● Packer ● kexec ● iPXE ● Write Boot Part ● Secure Boot ● ONIE ● Cloud Init ● BMC Boot ● Kickstart ● Preseed All roads lead to a kernel init process PXE PXE Let’s PXE! Bootstrapping is a multi-stage process Server DHCP Firmware PXE NextServer & Options TFTP Bootloader Stage 1 lpxelinux.0 Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi HTTP(S) O/S Kernel O/S ISO First: Get on the network Server DHCP Firmware PXE NextServer & Options Provisioning Service(s) Then: Download a Bootloader Server TFTP Bootloader Stage 1 lpxelinux.0 Provisioning Service(s) Then get a BETTER Bootloader Server Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi Finally load a “real” operating system Server Provisioning Service(s) HTTP(S) O/S Kernel O/S ISO Each stage is actually a NEW O/S Load Server DHCP Firmware DHCP PXE NextServer & Options TFTP Bootloader DHCP Stage 1 lpxelinux.0 Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi DHCP HTTP(S) DHCP O/S Kernel O/S ISO And modern servers can skip TFTP! So… technically, no longer PXE Server DHCP Firmware PXE NextServer & Options Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi HTTP(S) O/S Kernel O/S ISO Yay! We’re done, right? Provisioning is more than PXE Server iPXE O/S Kernel Bootloader O/S ISO Config Kickstart Templates Provisioning Service(s) Download Installation Packages Access & Post-Config Apps Hardware varies, so Install must be guided by templates Server O/S Kernel Config Kickstart Templates Provisioning Service(s) ISOs are minimal and stale So they must be updated Server O/S Kernel Kickstart Provisioning Service(s) Download Repo Installation Packages Mirrors And then you can actually connect to start configuring the system! Server O/S Kernel Kickstart Provisioning Service(s) Installation Access & Post-Config Apps Automating Provisioning means Connecting all these steps together Server iPXE O/S Kernel Bootloader O/S ISO Config Kickstart Templates Provisioning Service(s) Download Installation Packages Access & Post-Config Apps But wait…. There’s more to consider! Server iPXE Out of Band Management (BMC, IPMI, Redfish, etc) O/S Kernel Bootloader O/S ISO Infrastrastructure Config Kickstart as Code Templates Provisioning Service(s) Download Installation Packages Access & Post-Config Apps IaC? Show us some templates! Typical PXE Questions Why is this so fragile? What about PXE over Wifi? What about using a VLAN? Can I dockerize this? What about setting BIOS & RAID? How can I make this faster? How can we Server PXE/iPXE PXE simplify that?!! Small RAM only Footprint Provisioning O/S Inventory At RackN, we’ve been using an in Service(s) reboot memory operating system, “sledgehammer,” based on CentOS. Informed Guided Installation Installation It’s highly optimized to ● Run on nearly any hardware ● Load very quickly ● Collect deep inventory ● Have built-in tools for system tasks like hardware config Image Based Deployment (10x faster!) Server PXE/iPXE PXE RAM only Small O/S Footprint Provisioning Write O/S O/S Image Service(s) To Drive(s) as Archive reboot Informed Machine Installation Init Immutable Provisioning Server PXE PXE/iPXE Highly RAM only Minimal Available O/S Footprint Provisioning Service(s) Config & Machine Attach Disks Initialize Load Apps Container & Containers Registries And now… Advanced Provisioning! ESXi Provisioning Server iPXE O/S Kernel Bootloader O/S ISO Provisioning Config weasel Service(s) Templates restricted Access & CLI/python Apps Control via VMware ESXi VMw APIs Tooling ONIE: Open Network Install Environment Switch Current DHCP Firmware Provisioning Service(s) New HTTP(S) Firmware O/S Image Designed for Embedded Systems where we’re replacing the O/S as a complete image. Does have DHCP options for a startup script. kexec (kernel execute) Server Normal Running O/S Provision Download New Kernel New Kernel kexec Starts Provisioning Without New O/S Service(s) Rebooting kexec (kernel execute) Server Normal Running O/S Provision Download New Kernel New Kernel kexec Provisioning New O/S Service(s) Start kexec iPXE New Normal Installation Provision Without Rebooting Secure Boot Required SIGNED Bootloaders Server DHCP Secured Firmware NextServer & Options Enabled TPM HTTP(S) Trusted Provisioning Bootloader Signed ipxe.efi Service(s) Verified HTTP(S) Trusted O/S Kernel Signed O/S ISO Verified BMC Boot option 1 Server Install BMC Firmware Media Attached O/S Install Provisioning Service(s) Kickstart Download Installation Packages Post-Config Access & Apps BMC Boot option 2 Server Install BMC Firmware Media Bootloader DHCP Provisioning Config Service(s) Kickstart Templates Download Installation Packages Post-Config Access & Apps Thanks! Contact us: Rob Hirschfeld, RackN.com Digital Rebar Behind the Firewall, Self-Service Infrastructure as Code Self-Trials: rebar.digital .