Give Your PXE Wings! It’s not magic! How booting actually works.
Presentation for virtual SREcon 2020 By Rob Hirschfeld, RackN Rob Hirschfeld
@zehicle Co-Founder of RackN
We created Digital Rebar Bare Metal Provisioning ++
@L8istSh9y Podcast on PXE: http://bit.ly/pxewings In concept, O/SO/S Provisioning is Easy!
We’re just installing an operating system on a server or switch!
Why is that so hard?!
● Bootstrapping ● Firmware Limitations ● Variation ● Networking ● Security ● Performance Computer ● Post configuration In concept, Pre- & Provisioning is Easy! Post- Config We’re just installing an operating system on a server or switch!
Why is that so hard?!
● Bootstrapping ● Firmware Limitations And that’s not even including: ● Variation ● Networking ● System Inventory ● Security ● System Validation ● Performance ● Hardware Configuration ● Post configuration ● Naming & Addressing ● Credentials Injection Exploring Provisioning Approaches
Netboot (25 min) Image Deploy (10 min) Esoteric Flavors (5 min)
● PXE ● Packer ● kexec ● iPXE ● Write Boot Part ● Secure Boot ● ONIE ● Cloud Init ● BMC Boot ● Kickstart ● Preseed Exploring Provisioning Approaches
Netboot (25 min) Image Deploy (10 min) Esoteric Flavors (5 min)
● PXE ● Packer ● kexec ● iPXE ● Write Boot Part ● Secure Boot ● ONIE ● Cloud Init ● BMC Boot ● Kickstart ● Preseed
All roads lead to a kernel init process PXE PXE Let’s PXE! Bootstrapping is a multi-stage process
Server DHCP Firmware PXE NextServer & Options TFTP Bootloader Stage 1 lpxelinux.0 Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi
HTTP(S) O/S Kernel O/S ISO First: Get on the network
Server DHCP Firmware PXE NextServer & Options
Provisioning Service(s) Then: Download a Bootloader
Server
TFTP Bootloader Stage 1 lpxelinux.0 Provisioning Service(s) Then get a BETTER Bootloader
Server
Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi Finally load a “real” operating system
Server
Provisioning Service(s)
HTTP(S) O/S Kernel O/S ISO Each stage is actually a NEW O/S Load
Server DHCP Firmware DHCP PXE NextServer & Options TFTP Bootloader DHCP Stage 1 lpxelinux.0 Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi DHCP
HTTP(S) DHCP O/S Kernel O/S ISO And modern servers can skip TFTP! So… technically, no longer PXE
Server DHCP Firmware PXE NextServer & Options
Provisioning Service(s) HTTP(S) Bootloader Stage 2 ipxe.efi
HTTP(S) O/S Kernel O/S ISO Yay! We’re done, right? Provisioning is more than PXE
Server iPXE O/S Kernel Bootloader O/S ISO
Config Kickstart Templates Provisioning Service(s) Download Installation Packages
Access & Post-Config Apps Hardware varies, so Install must be guided by templates
Server
O/S Kernel
Config Kickstart Templates Provisioning Service(s) ISOs are minimal and stale So they must be updated
Server
O/S Kernel
Kickstart Provisioning Service(s) Download Repo Installation Packages Mirrors And then you can actually connect to start configuring the system!
Server
O/S Kernel
Kickstart Provisioning Service(s)
Installation
Access & Post-Config Apps
Automating Provisioning means Connecting all these steps together
Server iPXE O/S Kernel Bootloader O/S ISO
Config Kickstart Templates Provisioning Service(s) Download Installation Packages
Access & Post-Config Apps
But wait…. There’s more to consider!
Server iPXE Out of Band Management (BMC, IPMI, Redfish, etc) O/S Kernel Bootloader O/S ISO Infrastrastructure Config Kickstart as Code Templates Provisioning Service(s) Download Installation Packages
Access & Post-Config Apps IaC? Show us some templates! Typical PXE Questions
Why is this so fragile?
What about PXE over Wifi?
What about using a VLAN?
Can I dockerize this?
What about setting BIOS & RAID?
How can I make this faster? How can we Server PXE/iPXE PXE simplify that?!! Small RAM only Footprint Provisioning O/S Inventory At RackN, we’ve been using an in Service(s) reboot memory operating system, “sledgehammer,” based on CentOS. Informed Guided Installation Installation It’s highly optimized to
● Run on nearly any hardware ● Load very quickly ● Collect deep inventory ● Have built-in tools for system tasks like hardware config Image Based Deployment (10x faster!)
Server PXE/iPXE PXE
RAM only Small O/S Footprint Provisioning Write O/S O/S Image Service(s) To Drive(s) as Archive
reboot Informed Machine Installation Init Immutable Provisioning
Server
PXE PXE/iPXE Highly RAM only Minimal Available O/S Footprint Provisioning Service(s) Config & Machine Attach Disks Initialize
Load Apps Container & Containers Registries And now… Advanced Provisioning! ESXi Provisioning
Server iPXE O/S Kernel Bootloader O/S ISO Provisioning Config weasel Service(s) Templates
restricted Access & CLI/python Apps
Control via VMware ESXi VMw APIs Tooling ONIE: Open Network Install Environment
Switch
Current DHCP Firmware Provisioning Service(s) New HTTP(S) Firmware O/S Image
Designed for Embedded Systems where we’re replacing the O/S as a complete image.
Does have DHCP options for a startup script. kexec (kernel execute)
Server Normal Running O/S Provision Download New Kernel New Kernel kexec Starts Provisioning Without New O/S Service(s) Rebooting kexec (kernel execute)
Server Normal Running O/S Provision Download New Kernel New Kernel kexec Provisioning New O/S Service(s) Start kexec iPXE New Normal Installation Provision Without Rebooting Secure Boot Required SIGNED Bootloaders
Server DHCP Secured Firmware NextServer & Options Enabled TPM HTTP(S) Trusted Provisioning Bootloader Signed ipxe.efi Service(s) Verified HTTP(S) Trusted O/S Kernel Signed O/S ISO Verified BMC Boot option 1
Server Install BMC Firmware Media
Attached O/S Install Provisioning Service(s) Kickstart Download Installation Packages Post-Config Access & Apps BMC Boot option 2
Server Install BMC Firmware Media
Bootloader DHCP Provisioning Config Service(s) Kickstart Templates Download Installation Packages Post-Config Access & Apps Thanks!
Contact us:
Rob Hirschfeld, RackN.com
Digital Rebar Behind the Firewall, Self-Service Infrastructure as Code
Self-Trials: rebar.digital