Netflix and FreeBSD: Using Open Source to Deliver Streaming Video
Jonathan Looney FOSDEM 2019 Open Connect
NETFLIX AND FREEBSD FOSDEM 2019 Open Connect is Netflix’s CDN. It is global, efficient, and purpose-built for distributing Netflix’s content.
NETFLIX AND FREEBSD FOSDEM 2019 The Open Connect Appliance
The OCA is the “backbone” of the Open Connect network.
The OCA almost exclusively runs open-source software.
40Gb/s Storage Appliance with 248TB storage
NETFLIX AND FREEBSD (2RU form factor) FOSDEM 2019 Open Connect Traffic
Video Video Apps Apps Open Connect
Video Video Apps Apps
NETFLIX AND FREEBSD FOSDEM 2019 Netflix OCA Workload
RAM Plain-text Encrypted Data Data
CPU Disks NICs
NETFLIX AND FREEBSD FOSDEM 2019 Using FreeBSD and commodity parts, we achieve 90 Gb/s serving TLS-encrypted connections with ~55% CPU on a 16-core 2.6-GHz
CPU.
NETFLIX AND FREEBSD FOSDEM 2019 NETFLIX AND FREEBSD FOSDEM 2019 OCA Operating System
NETFLIX AND FREEBSD FOSDEM 2019 (Abridged) BSD “Family Tree”
AT&T Unix
BSD
NetBSD FreeBSD
NETFLIX AND FREEBSD FOSDEM 2019 FreeBSD Release Cycle
11.0 11.1 11.2
stable/11 12.0 12.1
stable/12 ? head
NETFLIX AND FREEBSD FOSDEM 2019 We Track FreeBSD “Head”
FreeBSD head
Netflix master
Netflix release branches
NETFLIX AND FREEBSD FOSDEM 2019 Typical Release Cycle
Testing
Feature Development/Integration
FreeBSD Dev Canary Phased Rollout Merge Testing Testing
Five Weeks of Development Five Weeks of Testing/Deployment
NETFLIX AND FREEBSD FOSDEM 2019 Examples of Features
● NUMA enhancements ● Asynchronous sendfile ● Kernel TLS ● Pbuf allocation enhancements ● “Unmapped” mbufs ● I/O scheduling ● TCP algorithms ● TCP logging infrastructure
NETFLIX AND FREEBSD FOSDEM 2019 Tracking “head” lets us stay forward looking and focused on innovation.
NETFLIX AND FREEBSD FOSDEM 2019 Downstream users of open-source projects can be stuck in “vicious” or “virtuous” cycles.
NETFLIX AND FREEBSD FOSDEM 2019 Vicious Cycle Infrequent Merges
Slower Many Feature Conflicts/ Velocity Regressions
NETFLIX AND FREEBSD FOSDEM 2019 Virtuous Cycle Frequent Merges
Faster Few Feature Conflicts/ Velocity/ Regressions Collaboration
NETFLIX AND FREEBSD FOSDEM 2019 Reasons We Keep Local Diffs
● Information covered under NDA ● Feature which is still in development/testing ● Feature which needs to be generalized
NETFLIX AND FREEBSD FOSDEM 2019 It is our intention to upstream any code which we can.
NETFLIX AND FREEBSD FOSDEM 2019 Benefits to Netflix of Tracking FreeBSD “Head”
● Quicker feature iteration ● Quicker access to new FreeBSD features ● Quicker bug fixes ● Enables collaboration ● Minimizes merge conflicts ● Amortizes merge “cost”
NETFLIX AND FREEBSD FOSDEM 2019 Benefits to FreeBSD
● Wide deployment of “head” branch code (albeit in a narrow use case) ● Early intensive testing ● Incentive for Netflix to upstream code
NETFLIX AND FREEBSD FOSDEM 2019 Objections to Running “Development” Code
● It isn’t stable ● Why should you pay to find the bugs others will find while testing head? ● Aren’t there more security bugs? ● No one runs development branches ● Pay monthly “cost” to do merges ● You get new bugs each month
NETFLIX AND FREEBSD FOSDEM 2019 Running FreeBSD “head” lets us deliver large amounts of data to our users very efficiently, while maintaining a high velocity of feature development.
NETFLIX AND FREEBSD FOSDEM 2019 Thank you