FreeBSD SMPng: Behind the scenes GregLehey [email protected] [email protected] Sydney, 27September 2001 FreeBSD SMPng behind the scenes 1GregLehey, 27September 2001 Topics • Howwegot into this mess. FreeBSD SMPng behind the scenes 2GregLehey, 27September 2001 Topics • Howwegot into this mess. • Whythe UNIX kernel is not suited to multiple processors. FreeBSD SMPng behind the scenes 3GregLehey, 27September 2001 Topics • Howwegot into this mess. • Whythe UNIX kernel is not suited to multiple processors. • Solving the problem. FreeBSD SMPng behind the scenes 4GregLehey, 27September 2001 Topics • Howwegot into this mess. • Whythe UNIX kernel is not suited to multiple processors. • Solving the problem. • Team dynamics. FreeBSD SMPng behind the scenes 5GregLehey, 27September 2001 Topics • Howwegot into this mess. • Whythe UNIX kernel is not suited to multiple processors. • Solving the problem. • Team dynamics. • Current state of play. FreeBSD SMPng behind the scenes 6GregLehey, 27September 2001 Topics • Howwegot into this mess. • Whythe UNIX kernel is not suited to multiple processors. • Solving the problem. • Team dynamics. • Current state of play. • Looking forward. FreeBSD SMPng behind the scenes 7•GregLehey, 27September 2001 The Mindcraft benchmarks • Common knowledge: UNIX is faster than Microsoft. FreeBSD SMPng behind the scenes 8GregLehey, 27September 2001 The Mindcraft benchmarks • Common knowledge: UNIX is faster than Microsoft. • In 1999, Mindcraft published benchmarks showing NT much faster than Linux. FreeBSD SMPng behind the scenes 9GregLehey, 27September 2001 The Mindcraft benchmarks • Common knowledge: UNIX is faster than Microsoft. • In 1999, Mindcraft published benchmarks showing NT much faster than Linux. • Linux people first claimed the results were wrong. FreeBSD SMPng behind the scenes 10 GregLehey, 27September 2001 The Mindcraft benchmarks • Common knowledge: UNIX is faster than Microsoft. • In 1999, Mindcraft published benchmarks showing NT much faster than Linux. • Linux people first claimed the results were wrong. • Linux people later realised the results were correct, but the benchmark was contrived. FreeBSD SMPng behind the scenes 11 GregLehey, 27September 2001 The Mindcraft benchmarks • Common knowledge: UNIX is faster than Microsoft. • In 1999, Mindcraft published benchmarks showing NT much faster than Linux. • Linux people first claimed the results were wrong. • Linux people later realised the results were correct, but the benchmark was contrived. • FreeBSD people kept very quiet. FreeBSD SMPng behind the scenes 12 GregLehey, 27September 2001 The Mindcraft benchmarks • Common knowledge: UNIX is faster than Microsoft. • In 1999, Mindcraft published benchmarks showing NT much faster than Linux. • Linux people first claimed the results were wrong. • Linux people later realised the results were correct, but the benchmark was contrived. • FreeBSD people kept very quiet. • One of the problems was the ‘‘big kernel lock’’SMP implementation. FreeBSD SMPng behind the scenes 13 • GregLehey, 27September 2001 The UNIX kernel design • One CPU FreeBSD SMPng behind the scenes 14 GregLehey, 27September 2001 The UNIX kernel design • One CPU • Processes perform user functions. FreeBSD SMPng behind the scenes 15 GregLehey, 27September 2001 The UNIX kernel design • One CPU • Processes perform user functions. • Interrupt handlers handle I/O. FreeBSD SMPng behind the scenes 16 GregLehey, 27September 2001 The UNIX kernel design • One CPU • Processes perform user functions. • Interrupt handlers handle I/O. • Interrupt handlers have priority overprocesses. FreeBSD SMPng behind the scenes 17 • GregLehey, 27September 2001 Processes • One CPU FreeBSD SMPng behind the scenes 18 GregLehey, 27September 2001 Processes • One CPU • Processes have different priorities. FreeBSD SMPng behind the scenes 19 GregLehey, 27September 2001 Processes • One CPU • Processes have different priorities. • The scheduler chooses the highest priority process which is ready to run. FreeBSD SMPng behind the scenes 20 GregLehey, 27September 2001 Processes • One CPU • Processes have different priorities. • The scheduler chooses the highest priority process which is ready to run. • The process can relinquish the CPU voluntarily (tsleep). FreeBSD SMPng behind the scenes 21 GregLehey, 27September 2001 Processes • One CPU • Processes have different priorities. • The scheduler chooses the highest priority process which is ready to run. • The process can relinquish the CPU voluntarily (tsleep). • The scheduler runs when the process finishes its time slice. FreeBSD SMPng behind the scenes 22 GregLehey, 27September 2001 Processes • One CPU • Processes have different priorities. • The scheduler chooses the highest priority process which is ready to run. • The process can relinquish the CPU voluntarily (tsleep). • The scheduler runs when the process finishes its time slice. • Processes are not scheduled while running kernel code. FreeBSD SMPng behind the scenes 23 • GregLehey, 27September 2001 Interrupts • Interrupts cannot be delayed until kernel is inactive. FreeBSD SMPng behind the scenes 24 GregLehey, 27September 2001 Interrupts • Interrupts cannot be delayed until kernel is inactive. • Different synchronization: block interrupts in critical kernel code. FreeBSD SMPng behind the scenes 25 GregLehey, 27September 2001 Interrupts • Interrupts cannot be delayed until kernel is inactive. • Different synchronization: block interrupts in critical kernel code. • Finer grained locking: splbio for block I/O, spltty for serial I/O, splnet for network devices, etc. FreeBSD SMPng behind the scenes 26 • GregLehey, 27September 2001 Problems on SMP machines • UNIX approach doesn’tallowmore than one process run- ning in kernel mode. FreeBSD SMPng behind the scenes 27 GregLehey, 27September 2001 Problems on SMP machines • UNIX approach doesn’tallowmore than one process run- ning in kernel mode. • ‘‘Solution’’: introduce Big Kernel Lock. Spin (loop) waiting for this lock if it’staken. FreeBSD SMPng behind the scenes 28 GregLehey, 27September 2001 Problems on SMP machines • UNIX approach doesn’tallowmore than one process run- ning in kernel mode. • ‘‘Solution’’: introduce Big Kernel Lock. Spin (loop) waiting for this lock if it’staken. • Disadvantage: much CPU time may be lost. FreeBSD SMPng behind the scenes 29 • GregLehey, 27September 2001 Getting out of the mess • Linux people started working very quickly. FreeBSD SMPng behind the scenes 30 GregLehey, 27September 2001 Getting out of the mess • Linux people started working very quickly. • FreeBSD people stunned, unable to move. FreeBSD SMPng behind the scenes 31 GregLehey, 27September 2001 Getting out of the mess • Linux people started working very quickly. • FreeBSD people stunned, unable to move. • FreeBSD people looked at the Linux solution and didn’tun- derstand FreeBSD SMPng behind the scenes 32 GregLehey, 27September 2001 Getting out of the mess • Linux people started working very quickly. • FreeBSD people stunned, unable to move. • FreeBSD people looked at the Linux solution and didn’t underˆHˆHˆHˆHˆHlikeit. FreeBSD SMPng behind the scenes 33 GregLehey, 27September 2001 Getting out of the mess • Linux people started working very quickly. • FreeBSD people stunned, unable to move. • FreeBSD people looked at the Linux solution and didn’t underˆHˆHˆHˆHˆHlikeit. • BSDi bought out Walnut Creek CDROM. FreeBSD SMPng behind the scenes 34 • GregLehey, 27September 2001 The BSD/OS merge • BSDi (previously BSDI) was the vendor of BSD/OS. FreeBSD SMPng behind the scenes 35 GregLehey, 27September 2001 The BSD/OS merge • BSDi (previously BSDI) was the vendor of BSD/OS. • Initial plans were to merge BSD/OS and FreeBSD. FreeBSD SMPng behind the scenes 36 GregLehey, 27September 2001 The BSD/OS merge • BSDi (previously BSDI) was the vendor of BSD/OS. • Initial plans were to merge BSD/OS and FreeBSD. • Licensing and technical issues stopped this merge. FreeBSD SMPng behind the scenes 37 GregLehey, 27September 2001 The BSD/OS merge • BSDi (previously BSDI) was the vendor of BSD/OS. • Initial plans were to merge BSD/OS and FreeBSD. • Licensing and technical issues stopped this merge. • BSD/OS already had better SMP support under develop- ment, code name ‘‘SMPng’’. FreeBSD SMPng behind the scenes 38 GregLehey, 27September 2001 The BSD/OS merge • BSDi (previously BSDI) was the vendor of BSD/OS. • Initial plans were to merge BSD/OS and FreeBSD. • Licensing and technical issues stopped this merge. • BSD/OS already had better SMP support under develop- ment, code name ‘‘SMPng’’. • FreeBSD developers were givenaccess to BSD/OS source code. FreeBSD SMPng behind the scenes 39 GregLehey, 27September 2001 The BSD/OS merge • BSDi (previously BSDI) was the vendor of BSD/OS. • Initial plans were to merge BSD/OS and FreeBSD. • Licensing and technical issues stopped this merge. • BSD/OS already had better SMP support under develop- ment, code name ‘‘SMPng’’. • FreeBSD developers were givenaccess to BSD/OS source code. • FreeBSD developers allowed to merge ‘‘significant parts’’ofBSD/OS code. FreeBSD SMPng behind the scenes 40 GregLehey, 27September 2001 The BSD/OS merge • BSDi (previously BSDI) was the vendor of BSD/OS. • Initial plans were to merge BSD/OS and FreeBSD. • Licensing and technical issues stopped this merge. • BSD/OS already had better SMP support under develop- ment, code name ‘‘SMPng’’. • FreeBSD developers were givenaccess to BSD/OS source code. • FreeBSD developers allowed to merge ‘‘significant parts’’ofBSD/OS code. • Decision made to merge SMPng.