Freebsd Smpng: Behind the Scenes

Home , FreeBSD

FreeBSD SMPng: Behind the scenes

GregLehey [email protected] [email protected] Sydney, 27September 2001

FreeBSD SMPng behind the scenes 1GregLehey, 27September 2001 Topics

• Howwegot into this mess.

FreeBSD SMPng behind the scenes 2GregLehey, 27September 2001 Topics

• Howwegot into this mess. • Whythe UNIX kernel is not suited to multiple processors.

FreeBSD SMPng behind the scenes 3GregLehey, 27September 2001 Topics

• Howwegot into this mess. • Whythe UNIX kernel is not suited to multiple processors. • Solving the problem.

FreeBSD SMPng behind the scenes 4GregLehey, 27September 2001 Topics

• Howwegot into this mess. • Whythe UNIX kernel is not suited to multiple processors. • Solving the problem. • Team dynamics.

FreeBSD SMPng behind the scenes 5GregLehey, 27September 2001 Topics

• Howwegot into this mess. • Whythe UNIX kernel is not suited to multiple processors. • Solving the problem. • Team dynamics. • Current state of play.

FreeBSD SMPng behind the scenes 6GregLehey, 27September 2001 Topics

• Howwegot into this mess. • Whythe UNIX kernel is not suited to multiple processors. • Solving the problem. • Team dynamics. • Current state of play. • Looking forward.

FreeBSD SMPng behind the scenes 7•GregLehey, 27September 2001 The Mindcraft benchmarks

• Common knowledge: UNIX is faster than Microsoft.

FreeBSD SMPng behind the scenes 8GregLehey, 27September 2001 The Mindcraft benchmarks

• Common knowledge: UNIX is faster than Microsoft. • In 1999, Mindcraft published benchmarks showing NT much faster than Linux.

FreeBSD SMPng behind the scenes 9GregLehey, 27September 2001 The Mindcraft benchmarks

• Common knowledge: UNIX is faster than Microsoft. • In 1999, Mindcraft published benchmarks showing NT much faster than Linux. • Linux people ﬁrst claimed the results were wrong.

FreeBSD SMPng behind the scenes 10 GregLehey, 27September 2001 The Mindcraft benchmarks

• Common knowledge: UNIX is faster than Microsoft. • In 1999, Mindcraft published benchmarks showing NT much faster than Linux. • Linux people ﬁrst claimed the results were wrong. • Linux people later realised the results were correct, but the benchmark was contrived.

FreeBSD SMPng behind the scenes 11 GregLehey, 27September 2001 The Mindcraft benchmarks

FreeBSD SMPng behind the scenes 12 GregLehey, 27September 2001 The Mindcraft benchmarks

FreeBSD SMPng behind the scenes 13 • GregLehey, 27September 2001 The UNIX kernel design

• One CPU

FreeBSD SMPng behind the scenes 14 GregLehey, 27September 2001 The UNIX kernel design

• One CPU • Processes perform user functions.

FreeBSD SMPng behind the scenes 15 GregLehey, 27September 2001 The UNIX kernel design

• One CPU • Processes perform user functions. • Interrupt handlers handle I/O.

FreeBSD SMPng behind the scenes 16 GregLehey, 27September 2001 The UNIX kernel design

• One CPU • Processes perform user functions. • Interrupt handlers handle I/O. • Interrupt handlers have priority overprocesses.

FreeBSD SMPng behind the scenes 17 • GregLehey, 27September 2001 Processes

• One CPU

FreeBSD SMPng behind the scenes 18 GregLehey, 27September 2001 Processes

• One CPU • Processes have different priorities.

FreeBSD SMPng behind the scenes 19 GregLehey, 27September 2001 Processes

• One CPU • Processes have different priorities. • The scheduler chooses the highest priority process which is ready to run.

FreeBSD SMPng behind the scenes 20 GregLehey, 27September 2001 Processes

• One CPU • Processes have different priorities. • The scheduler chooses the highest priority process which is ready to run. • The process can relinquish the CPU voluntarily (tsleep).

FreeBSD SMPng behind the scenes 21 GregLehey, 27September 2001 Processes

• One CPU • Processes have different priorities. • The scheduler chooses the highest priority process which is ready to run. • The process can relinquish the CPU voluntarily (tsleep). • The scheduler runs when the process ﬁnishes its time slice.

FreeBSD SMPng behind the scenes 22 GregLehey, 27September 2001 Processes

FreeBSD SMPng behind the scenes 23 • GregLehey, 27September 2001 Interrupts

• Interrupts cannot be delayed until kernel is inactive.

FreeBSD SMPng behind the scenes 24 GregLehey, 27September 2001 Interrupts

• Interrupts cannot be delayed until kernel is inactive. • Different synchronization: block interrupts in critical kernel code.

FreeBSD SMPng behind the scenes 25 GregLehey, 27September 2001 Interrupts

• Interrupts cannot be delayed until kernel is inactive. • Different synchronization: block interrupts in critical kernel code. • Finer grained locking: splbio for block I/O, spltty for serial I/O, splnet for network devices, etc.

FreeBSD SMPng behind the scenes 26 • GregLehey, 27September 2001 Problems on SMP machines

• UNIX approach doesn’tallowmore than one process run- ning in kernel mode.

FreeBSD SMPng behind the scenes 27 GregLehey, 27September 2001 Problems on SMP machines

• UNIX approach doesn’tallowmore than one process run- ning in kernel mode. • ‘‘Solution’’: introduce Big Kernel Lock. Spin (loop) waiting for this lock if it’staken.

FreeBSD SMPng behind the scenes 28 GregLehey, 27September 2001 Problems on SMP machines

• UNIX approach doesn’tallowmore than one process run- ning in kernel mode. • ‘‘Solution’’: introduce Big Kernel Lock. Spin (loop) waiting for this lock if it’staken. • Disadvantage: much CPU time may be lost.

FreeBSD SMPng behind the scenes 29 • GregLehey, 27September 2001 Getting out of the mess

• Linux people started working very quickly.

FreeBSD SMPng behind the scenes 30 GregLehey, 27September 2001 Getting out of the mess

• Linux people started working very quickly. • FreeBSD people stunned, unable to move.

FreeBSD SMPng behind the scenes 31 GregLehey, 27September 2001 Getting out of the mess

• Linux people started working very quickly. • FreeBSD people stunned, unable to move. • FreeBSD people looked at the Linux solution and didn’tun- derstand

FreeBSD SMPng behind the scenes 32 GregLehey, 27September 2001 Getting out of the mess

• Linux people started working very quickly. • FreeBSD people stunned, unable to move. • FreeBSD people looked at the Linux solution and didn’t underˆHˆHˆHˆHˆHlikeit.

FreeBSD SMPng behind the scenes 33 GregLehey, 27September 2001 Getting out of the mess

• Linux people started working very quickly. • FreeBSD people stunned, unable to move. • FreeBSD people looked at the Linux solution and didn’t underˆHˆHˆHˆHˆHlikeit. • BSDi bought out Walnut Creek CDROM.

FreeBSD SMPng behind the scenes 34 • GregLehey, 27September 2001 The BSD/OS merge

• BSDi (previously BSDI) was the vendor of BSD/OS.

FreeBSD SMPng behind the scenes 35 GregLehey, 27September 2001 The BSD/OS merge

• BSDi (previously BSDI) was the vendor of BSD/OS. • Initial plans were to merge BSD/OS and FreeBSD.

FreeBSD SMPng behind the scenes 36 GregLehey, 27September 2001 The BSD/OS merge

• BSDi (previously BSDI) was the vendor of BSD/OS. • Initial plans were to merge BSD/OS and FreeBSD. • Licensing and technical issues stopped this merge.

FreeBSD SMPng behind the scenes 37 GregLehey, 27September 2001 The BSD/OS merge

FreeBSD SMPng behind the scenes 38 GregLehey, 27September 2001 The BSD/OS merge

• BSDi (previously BSDI) was the vendor of BSD/OS. • Initial plans were to merge BSD/OS and FreeBSD. • Licensing and technical issues stopped this merge. • BSD/OS already had better SMP support under development, code name ‘‘SMPng’’. • FreeBSD developers were givenaccess to BSD/OS source code.

FreeBSD SMPng behind the scenes 39 GregLehey, 27September 2001 The BSD/OS merge

FreeBSD SMPng behind the scenes 40 GregLehey, 27September 2001 The BSD/OS merge

FreeBSD SMPng behind the scenes 41 • GregLehey, 27September 2001 The meeting at Yahoo!

• Interested developers met at Yahoo! in June 2000. • Total of 20 participants. • 11 FreeBSD developers. • 3Apple developers. • 3Yahoo! staff. • 2BSDi developers.

FreeBSD SMPng behind the scenes 42 • GregLehey, 27September 2001 Limiting the delays

• Create ‘‘ﬁne-grained’’locking: lock only small parts of the kernel.

FreeBSD SMPng behind the scenes 43 GregLehey, 27September 2001 Limiting the delays

• Create ‘‘ﬁne-grained’’locking: lock only small parts of the kernel. • If resource is not available, block, don’tspin.

FreeBSD SMPng behind the scenes 44 GregLehey, 27September 2001 Limiting the delays

• Create ‘‘ﬁne-grained’’locking: lock only small parts of the kernel. • If resource is not available, block, don’tspin. • Problem: interrupt handlers can’tblock.

FreeBSD SMPng behind the scenes 45 GregLehey, 27September 2001 Limiting the delays

• Create ‘‘ﬁne-grained’’locking: lock only small parts of the kernel. • If resource is not available, block, don’tspin. • Problem: interrupt handlers can’tblock. • Solution: let them block, then.

FreeBSD SMPng behind the scenes 46 • GregLehey, 27September 2001 Blocking interrupt handlers

• Interrupt handlers get a process context.

FreeBSD SMPng behind the scenes 47 GregLehey, 27September 2001 Blocking interrupt handlers

• Interrupt handlers get a process context. • Short term: normal processes, involvescheduler overhead on ev ery invocation.

FreeBSD SMPng behind the scenes 48 GregLehey, 27September 2001 Blocking interrupt handlers

FreeBSD SMPng behind the scenes 49 GregLehey, 27September 2001 Blocking interrupt handlers

• Interrupt handlers get a process context. • Short term: normal processes, involvescheduler overhead on ev ery invocation. • Longer term: ‘‘light weight interrupt threads’’, scheduled only when conﬂicts occur. • Choice dictated by stability requirements during changeover.

FreeBSD SMPng behind the scenes 50 GregLehey, 27September 2001 Blocking interrupt handlers

FreeBSD SMPng behind the scenes 51 • GregLehey, 27September 2001 Blocking interrupt handlers

USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 10 99.0 0.0 00?? RL Thu10AM 4332:57.28 (idle: cpu1) root 11 99.0 0.0 00?? RL Thu10AM 4331:26.95 (idle: cpu0) root 12 0.0 0.0 00?? WL Thu10AM 6:02.04 (swi1: net) root 13 0.0 0.0 00?? WL Thu10AM 11:43.06 (swi6: tty:sio clock) root 14 0.0 0.0 00?? WL Thu10AM 0:00.00 (swi4: vm) root 15 0.0 0.0 00?? WL Thu10AM 0:00.00 (swi2: camnet) root 16 0.0 0.0 00?? WL Thu10AM 0:00.00 (swi3: cambio) root 17 0.0 0.0 00?? WL Thu10AM 0:00.00 (swi5: task queue) root 18 0.0 0.0 00?? WL Thu10AM 0:00.16 (irq14: ata0) root 19 0.0 0.0 00?? WL Thu10AM 0:00.00 (irq15: ata1) root 20 0.0 0.0 00?? WL Thu10AM 10:38.31 (irq9: dc0) root 21 0.0 0.0 00?? WL Thu10AM 1:51.00 (irq11: atapci1+) root 22 0.0 0.0 00?? WL Thu10AM 0:00.00 (irq1: atkbd0) root 23 0.0 0.0 00?? WL Thu10AM 0:00.00 (swi0: tty:sio) root 24 0.0 0.0 00?? WL Thu10AM 0:00.00 (irq4: sio0) root 25 0.0 0.0 00?? WL Thu10AM 0:00.00 (irq7: ppc0) root 26 0.0 0.0 00?? WL Thu10AM 0:00.00 (irq0: clk) root 27 0.0 0.0 00?? WL Thu10AM 0:00.00 (irq8: rtc) root 2 0.0 0.0 00?? DL Thu10AM 0:02.33 (pagedaemon) root 3 0.0 0.0 00?? DL Thu10AM 0:00.00 (vmdaemon) root 4 0.0 0.0 00?? DL Thu10AM 0:00.02 (pagezero) root 5 0.0 0.0 00?? DL Thu10AM 0:08.08 (bufdaemon) root 6 0.0 0.0 00?? DL Thu10AM 2:17.24 (syncer) root 0 0.0 0.0 00?? DLs Thu10AM 0:01.66 (swapper) root 1 0.0 0.1 656 29 ?? ILs Thu10AM 0:03.93 /sbin/init -d

FreeBSD SMPng behind the scenes 52 GregLehey, 27September 2001 Conclusions at Yahoo!

• Aim for stability,not performance, during the development.

FreeBSD SMPng behind the scenes 53 GregLehey, 27September 2001 Conclusions at Yahoo!

• Aim for stability,not performance, during the development. • Port BSDi code, don’treinvent the wheel.

FreeBSD SMPng behind the scenes 54 GregLehey, 27September 2001 Conclusions at Yahoo!

• Aim for stability,not performance, during the development. • Port BSDi code, don’treinvent the wheel. • Use heavy-weight threads at ﬁrst to enable better debugging.

FreeBSD SMPng behind the scenes 55 • GregLehey, 27September 2001 Responsibilities

• Matt Dillon to port locking primitivesand schedlock.

FreeBSD SMPng behind the scenes 56 GregLehey, 27September 2001 Responsibilities

• Matt Dillon to port locking primitivesand schedlock. • GregLeheytoport interrupt threads.

FreeBSD SMPng behind the scenes 57 GregLehey, 27September 2001 Responsibilities

• Matt Dillon to port locking primitivesand schedlock. • GregLeheytoport interrupt threads. • Doug Rabson to handle Alpha issues.

FreeBSD SMPng behind the scenes 58 GregLehey, 27September 2001 Responsibilities

• Matt Dillon to port locking primitivesand schedlock. • GregLeheytoport interrupt threads. • Doug Rabson to handle Alpha issues. • Paul Saab to enhance ddb.

FreeBSD SMPng behind the scenes 59 GregLehey, 27September 2001 Responsibilities

FreeBSD SMPng behind the scenes 60 GregLehey, 27September 2001 Responsibilities

• Matt Dillon to port locking primitivesand schedlock. • GregLeheytoport interrupt threads. • Doug Rabson to handle Alpha issues. • Paul Saab to enhance ddb. • Jonathan Lemon to convert network drivers. • Chuck Paterson of BSDi to be project liaison.

FreeBSD SMPng behind the scenes 61 GregLehey, 27September 2001 Responsibilities

FreeBSD SMPng behind the scenes 62 GregLehey, 27September 2001 Responsibilities

FreeBSD SMPng behind the scenes 63 • GregLehey, 27September 2001 First steps

• Matt Dillon had a tight deadline.

FreeBSD SMPng behind the scenes 64 GregLehey, 27September 2001 First steps

• Matt Dillon had a tight deadline. • Porting the BSD/OS code was more difﬁcult than anticipat- ed.

FreeBSD SMPng behind the scenes 65 GregLehey, 27September 2001 First steps

• Matt Dillon had a tight deadline. • Porting the BSD/OS code was more difﬁcult than anticipat- ed. • Matt rewrote the code.

FreeBSD SMPng behind the scenes 66 GregLehey, 27September 2001 First steps

• Matt Dillon had a tight deadline. • Porting the BSD/OS code was more difﬁcult than anticipat- ed. • Matt rewrote the code. • Code later ported by JakeBurkholder.

FreeBSD SMPng behind the scenes 67 • GregLehey, 27September 2001 First steps

• GregLeheystarted porting interrupt threads.

FreeBSD SMPng behind the scenes 68 GregLehey, 27September 2001 First steps

• GregLeheystarted porting interrupt threads. • Greghad twoweeks to port the code.

FreeBSD SMPng behind the scenes 69 GregLehey, 27September 2001 First steps

• GregLeheystarted porting interrupt threads. • Greghad twoweeks to port the code. • Porting the BSD/OS code was more difﬁcult than anticipat- ed.

FreeBSD SMPng behind the scenes 70 GregLehey, 27September 2001 First steps

• GregLeheystarted porting interrupt threads. • Greghad twoweeks to port the code. • Porting the BSD/OS code was more difﬁcult than anticipat- ed. • Heavyweight threads only in SPARC code.

FreeBSD SMPng behind the scenes 71 GregLehey, 27September 2001 First steps

• GregLeheystarted porting interrupt threads. • Greghad twoweeks to port the code. • Porting the BSD/OS code was more difﬁcult than anticipat- ed. • Heavyweight threads only in SPARC code. • SPARC and Intel code very different.

FreeBSD SMPng behind the scenes 72 • GregLehey, 27September 2001 First steps

• Differences between FreeBSD and BSD/OS larger than an- ticipated.

FreeBSD SMPng behind the scenes 73 GregLehey, 27September 2001 First steps

• Differences between FreeBSD and BSD/OS larger than an- ticipated. • Four-way comparison: FreeBSD 4.0, BSD/OS 4.0, BSD/OS 5Intel, BSD/OS 5 SPARC.

FreeBSD SMPng behind the scenes 74 GregLehey, 27September 2001 First steps

• Differences between FreeBSD and BSD/OS larger than an- ticipated. • Four-way comparison: FreeBSD 4.0, BSD/OS 4.0, BSD/OS 5Intel, BSD/OS 5 SPARC. • Took twomonths.

FreeBSD SMPng behind the scenes 75 • GregLehey, 27September 2001 Initial commit

From: Jason Evans To: [email protected], [email protected] Subject: cvs commit: src/bin/ps print.c src/share/man/man9 mutex.9 Makefile src/usr.bin/top machine.c src/sys/alpha/alpha mp_machdep.c synch_machdep.c clock.c genassym.c interrupt.c ipl_funcs.c locore.s machdep.c mem.c pmap.c prom.c support.s swtch.s trap.c ... jasone 2000/09/06 18:33:03 PDT (file names omitted) Log: Major update to the way synchronization is done in the kernel. Highlights include: *Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) *Per-CPU idle processes. *Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh

FreeBSD SMPng behind the scenes 76 GregLehey, 27September 2001 Opening the ﬂood gates

• After this point, manypeople got involved.

FreeBSD SMPng behind the scenes 77 GregLehey, 27September 2001 Opening the ﬂood gates

• After this point, manypeople got involved. • Much cosmetic work.

FreeBSD SMPng behind the scenes 78 GregLehey, 27September 2001 Opening the ﬂood gates

• After this point, manypeople got involved. • Much cosmetic work. • Terminology problems: what is a ‘‘mutex’’?

FreeBSD SMPng behind the scenes 79 GregLehey, 27September 2001 Opening the ﬂood gates

FreeBSD SMPng behind the scenes 80 GregLehey, 27September 2001 Opening the ﬂood gates

• After this point, manypeople got involved. • Much cosmetic work. • Terminology problems: what is a ‘‘mutex’’? • No clear direction: Manycontributors decided on their own locking constructs. • Loss of Jason Evans as project leader in March 2001.

FreeBSD SMPng behind the scenes 81 GregLehey, 27September 2001 Opening the ﬂood gates

FreeBSD SMPng behind the scenes 82 • GregLehey, 27September 2001 The kernel summit

• ‘‘Kernel summit’’held at Boston USENIX conference on Saturday,30June 2001.

FreeBSD SMPng behind the scenes 83 GregLehey, 27September 2001 The kernel summit

• ‘‘Kernel summit’’held at Boston USENIX conference on Saturday,30June 2001. • Manyinterruptions.

FreeBSD SMPng behind the scenes 84 GregLehey, 27September 2001 The kernel summit

• ‘‘Kernel summit’’held at Boston USENIX conference on Saturday,30June 2001. • Manyinterruptions. • Conclusion: release FreeBSD 5.0 in November 2001, ready or not.

FreeBSD SMPng behind the scenes 85 • GregLehey, 27September 2001 Project communications

• Mailing list [email protected],very little trafﬁc.

FreeBSD SMPng behind the scenes 86 GregLehey, 27September 2001 Project communications

• Mailing list [email protected],very little trafﬁc. • Much trafﬁc on IRC channel.

FreeBSD SMPng behind the scenes 87 GregLehey, 27September 2001 Project communications

• Mailing list [email protected],very little trafﬁc. • Much trafﬁc on IRC channel. • IRC tended to limit the number of participants.

FreeBSD SMPng behind the scenes 88 GregLehey, 27September 2001 Project communications

• Mailing list [email protected],very little trafﬁc. • Much trafﬁc on IRC channel. • IRC tended to limit the number of participants. • No record of discussions.

FreeBSD SMPng behind the scenes 89 • GregLehey, 27September 2001 Debugging tools

• BSD/OS supplied debugging tools.

FreeBSD SMPng behind the scenes 90 GregLehey, 27September 2001 Debugging tools

• BSD/OS supplied debugging tools. • Not all BSD/OS tools have been ported.

FreeBSD SMPng behind the scenes 91 GregLehey, 27September 2001 Debugging tools

• BSD/OS supplied debugging tools. • Not all BSD/OS tools have been ported. • Fewnew tools have been written: debugging is not interest- ing enough.

FreeBSD SMPng behind the scenes 92 GregLehey, 27September 2001 Debugging tools

• BSD/OS supplied debugging tools. • Not all BSD/OS tools have been ported. • Fewnew tools have been written: debugging is not interest- ing enough. • Good task for ‘‘Junior Kernel Hacker’’.

FreeBSD SMPng behind the scenes 93 • GregLehey, 27September 2001 Documentation

• Change logs very well documented.

FreeBSD SMPng behind the scenes 94 GregLehey, 27September 2001 Documentation

• Change logs very well documented. • Individual functions well documented.

FreeBSD SMPng behind the scenes 95 GregLehey, 27September 2001 Documentation

• Change logs very well documented. • Individual functions well documented. • Overall project plan less clear.

FreeBSD SMPng behind the scenes 96 • GregLehey, 27September 2001 Sample man page

MUTEX(9) FreeBSD Kernel Developer's Manual MUTEX(9) NAME mutex, mtx_init, mtx_lock, mtx_lock_spin, mtx_lock_flags, mtx_lock_spin_flags, mtx_trylock, mtx_trylock_flags, mtx_unlock, mtx_unlock_spin, mtx_unlock_flags, mtx_unlock_spin_flags, mtx_destroy, mtx_initialized, mtx_owned, mtx_recursed, mtx_assert - kernel synchronization primitives SYNOPSIS #include #include #include void mtx_init(struct mtx *mutex, const char *description, int opts); void mtx_lock(struct mtx *mutex); void mtx_lock_spin(struct mtx *mutex); void mtx_lock_flags(struct mtx *mutex, int flags);

FreeBSD SMPng behind the scenes 97 GregLehey, 27September 2001 Sample man page (continued)

DESCRIPTION Mutexes are the most basic and primary method of process synchronization. The major design considerations for mutexes are: 1. Acquiring and releasing uncontested mutexes should be as cheap as possible. 2. They must have the information and storage space to support priority propagation. 3. A process must be able to recursively acquire a mutex, provided that the mutex is initialized to support recursion. There are currently two flavors of mutexes, those that context switch when they block and those that do not. By default, MTX_DEF mutexes will context switch when they are already held. As amachine dependent optimization they may spin for some amount of time before context switching. It is important to remember that since aprocess may be preempted at any time, the possible context switch introduced by acquiring a mutex is guaranteed to not break anything that isn't already broken.

FreeBSD SMPng behind the scenes 98 GregLehey, 27September 2001 Sample commit log

jhb 2001/09/10 14:04:49 PDT Modified files: sys/kern kern_shutdown.c Log: -Axe holding_giant as it is not used now anyways and was ok'd by dillon in an earlier e-mail. -Wedon't need to test the console right before we vfprintf() the panicstr message. The printing of the panic message is a fine console test by itself and doesn't make useful messages scroll off the screen or tick developers off in quite the same. Requested by: jlemon, imp, bmilekic, chris, gsutter, jake (2) Revision Changes Path 1.110 +5 -36 src/sys/kern/kern_shutdown.c

FreeBSD SMPng behind the scenes 99 GregLehey, 27September 2001 Other commit logs

From: John Baldwin Date: Tue, 21 Nov 2000 13:10:15 -0800 (PST) jhb 2000/11/21 13:10:15 PST Modified files: sys/kern kern_ktr.c Log: Ahem, fix the disclaimer portion of the copyright so it disclaim's the voices in my head. You can sue the voices in Bill Paul's head all you want. Noticed by: jhb Revision Changes Path 1.6 +3 -3 src/sys/kern/kern_ktr.c

FreeBSD SMPng behind the scenes 100 GregLehey, 27September 2001 Other commit logs

--- kern_ktr.c 2000/11/15 21:51:53 1.5 +++ kern_ktr.c 2000/11/21 21:10:15 1.6 @@ -14,10 +14,10 @@ *may be used to endorse or promote products derived from this software *without specific prior written permission. * -*THIS SOFTWARE IS PROVIDED BY Bill Paul AND CONTRIBUTORS ``AS IS'' AND +*THIS SOFTWARE IS PROVIDED BY JOHN BALDWIN AND CONTRIBUTORS ``AS IS'' AND *ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE *IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -*ARE DISCLAIMED. IN NO EVENT SHALL Bill Paul OR THE VOICES IN HIS HEAD +*ARE DISCLAIMED. IN NO EVENT SHALL JOHN BALDWIN OR THE VOICES IN HIS HEAD *BELIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR *CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF *SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS @@ -26,7 +26,7 @@ *ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF *THE POSSIBILITY OF SUCH DAMAGE. * -*$FreeBSD: src/sys/kern/kern_ktr.c,v 1.5 2000/11/15 21:51:53 jhb Exp $ +*$FreeBSD: src/sys/kern/kern_ktr.c,v 1.6 2000/11/21 21:10:15 jhb Exp $ */ /*

FreeBSD SMPng behind the scenes 101 GregLehey, 27September 2001 Other commit logs

From: John Baldwin On 21-Nov-00 John Baldwin wrote: >jhb 2000/11/21 13:10:15 PST > >Modified files: >sys/kern kern_ktr.c >Log: >Ahem, fix the disclaimer portion of the copyright so it disclaim's the >voices in my head. You can sue the voices in Bill Paul's head all you >want. > >Noticed by: jhb Oh geez. That should be 'Noticed by: jlemon'. Iguess the voices are getting a bit too rambunctious.

FreeBSD SMPng behind the scenes 102 GregLehey, 27September 2001 Other commit logs

From: Warner Losh In message John Baldwin writes: :Ohgeez. That should be 'Noticed by: jlemon'. Iguess the voices :are getting a bit too rambunctious. It could be worse. You could be talking about yourself in the third person. Warner hates it when he does that.

FreeBSD SMPng behind the scenes 103 GregLehey, 27September 2001 Other commit logs

From: John Baldwin On 21-Nov-00 Warner Losh wrote: >Inmessage John Baldwin writes: >: Oh geez. That should be 'Noticed by: jlemon'. Iguess the voices are >: getting >: a bit too rambunctious. > >Itcould be worse. You could be talking about yourself in the third >person. Warner hates it when he does that. Well, I'm sure Warner will have a private discussion with Warner about doing that in public. Iwonder how the voices do their locking...

FreeBSD SMPng behind the scenes 104 GregLehey, 27September 2001 Other commit logs

From: Warner Losh In message John Baldwin writes: :On21-Nov-00 Warner Losh wrote: :>Inmessage John Baldwin writes: :>:Ohgeez. That should be 'Noticed by: jlemon'. Iguess the voices are :>:getting :>:abit too rambunctious. :> :>Itcould be worse. You could be talking about yourself in the third :>person. Warner hates it when he does that. : :Well, I'm sure Warner will have a private discussion with Warner about :doing that in public. Warner will do that only if Warner notices. :Iwonder how the voices do their locking... Warner Speculates that Warner's voices don't do locking.

FreeBSD SMPng behind the scenes 105 GregLehey, 27September 2001 Other commit logs

From: John Baldwin On 21-Nov-00 Warner Losh wrote: >Inmessage John Baldwin writes: >: Well, I'm sure Warner will have a private discussion with Warner about >: doing that in public. > >Warner will do that only if Warner notices. > >: I wonder how the voices do their locking... > >Warner Speculates that Warner's voices don't do locking. John thinJohn's voices are too inks that the consefficient to useole driver doesn't ha sleep locks and endve any lock up sping yeinning a lott. fatal double fault eip = 0x000000 ebp = %62F k epomn e

FreeBSD SMPng behind the scenes 106 GregLehey, 27September 2001 (untangled)

John thinJohn’svoices are too inks that the consefﬁcient to useole driverdoesn’thasleep locks and endveany lock up sping yeinning a lott. 1. John thinks that the console driverdoesn’thav e any locking yet. 2. John’svoices are too efﬁcient to use sleep locks and end up spinning a lot.

FreeBSD SMPng behind the scenes 107 GregLehey, 27September 2001 Performance

• Initial performance drop was expected due to scheduling interrupts.

FreeBSD SMPng behind the scenes 108 GregLehey, 27September 2001 Performance

• Initial performance drop was expected due to scheduling interrupts. • Final implementation should be much better (main project aim).

FreeBSD SMPng behind the scenes 109 GregLehey, 27September 2001 Performance

• Initial performance drop was expected due to scheduling interrupts. • Final implementation should be much better (main project aim). • What effect does this have onsingle processor systems?

FreeBSD SMPng behind the scenes 110 GregLehey, 27September 2001 Performance

FreeBSD SMPng behind the scenes 111 GregLehey, 27September 2001 Performance

• Initial performance drop was expected due to scheduling interrupts. • Final implementation should be much better (main project aim). • What effect does this have onsingle processor systems? • Currently we don’tknowhow good it will be. • Howdoweaddress the ‘‘howmanyproces- sors?’’question?

FreeBSD SMPng behind the scenes 112 GregLehey, 27September 2001 Performance

FreeBSD SMPng behind the scenes 113 GregLehey, 27September 2001 Performance

FreeBSD SMPng behind the scenes 114 • GregLehey, 27September 2001 Stability

• Prime goal in the short term: sacriﬁce performance for stability.

FreeBSD SMPng behind the scenes 115 GregLehey, 27September 2001 Stability

• Prime goal in the short term: sacriﬁce performance for stability. • Only partially successful.

FreeBSD SMPng behind the scenes 116 GregLehey, 27September 2001 Stability

• Prime goal in the short term: sacriﬁce performance for stability. • Only partially successful. • Manydev elopers no longer run -CURRENT.

FreeBSD SMPng behind the scenes 117 GregLehey, 27September 2001 Stability

• Prime goal in the short term: sacriﬁce performance for stability. • Only partially successful. • Manydev elopers no longer run -CURRENT. • Stability strongly dependent on hardware in use.

FreeBSD SMPng behind the scenes 118 • GregLehey, 27September 2001 What a difference a month makes

Paper submitted in August. Since then, • Matt Dillon ‘‘pushes down’’system call locking.

FreeBSD SMPng behind the scenes 119 GregLehey, 27September 2001 What a difference a month makes

Paper submitted in August. Since then, • Matt Dillon ‘‘pushes down’’system call locking. • Julian Elischer introduces kernel threads.

FreeBSD SMPng behind the scenes 120 GregLehey, 27September 2001 What a difference a month makes

Paper submitted in August. Since then, • Matt Dillon ‘‘pushes down’’system call locking. • Julian Elischer introduces kernel threads. • The release team decides to postpone the release by one year.

FreeBSD SMPng behind the scenes 121 • GregLehey, 27September 2001 Further information http://www.FreeBSD.org/smp/ Join in! The FreeBSD project needs more cleverhackers. These slides are available at http://www.lemis.com/SMPng/AUUG2001/

FreeBSD SMPng behind the scenes 122 GregLehey, 27September 2001