Introduction FreeBSD: Research results: incremental development " democracy can be very efficient (delegation of authority) and democratic organization " "see bug, fix bug, see bug fixed" is highly motivating

Why FreeBSD ? " there are other projects than + Apache ! Niels Jørgensen " Danish members of "core team" (2000) Roskilde University Center " [email protected] not because of FreeBSD democracy (I didn’t know) Open source / democracy Report: www.ruc.dk/~nielsj/research//freebsd.pdf " source available, license allows use,copy, modify Data: www.ruc.dk/~nielsj/research/freebsd/freebsd.html Slides: www.ruc.dk/~nielsj/research/freebsd/slides−dm−10−01−02.pdf " limited financial resources, "alternative" organization " but not necessarily "flat" organization

Outline What is FreeBSD ?

1. Intro Unix−type 2. Incremental development " very similar to Linux " runs 15% of Internet−servers worldwide (Yahoo, Hotmail, ..) 3. FreeBSD background Berkely Software Distribution 1977− approx. 90: 4. FreeBSD’s "Life cycle for changes" " key role in early Unix, Internet, and ”Open Source” history " good starting point in 90 (when given free by Berkeley U.) 5. Case study of development of major new feature FreeBSD / Linux non−technical differences 6. Conclusion " elitist image vs. anarchy, tolerance, young nerds, .. " lawsuit vs. early 90s jump start Note: " copy centrism vs. copy leftism " " generic sw development perspective FreeBSD license allows companies to modify " nothing about DM’s IT−network initiative and re−distribute (not "infecting" as GNU license) FreeBSD / Linux technical differences FreeBSD / Linux merge ?

Full distribution vs. pure kernel (Not that it is a real issue, but it is tempting to speculate) " 4000+ pieces of third party software " 50+ Java−related ports (compilers, JVMs, JIT−compilers) Pros: " stronger alternative to MS Windows (46%+ of Internet servers) Internet server vs. desktop/laptop " manpower to smoothen installation/configuration " project focus on security (eg. security auditing of ports) " fewer drivers Cons: " alternative ideas can be tried out in practice I don’t know whether FreeBSD’s − " waste of resources to fight over project decisions " virtual memory system is more suited for servers ? " software is shared today (eg. LinuxThreads on FreeBSD) " kernel is more stable ? " not a constant pool of programmes " double manpower not same as double output Otherwise: " similar basic principles ("Unix−type") " shares software for graphical interface, shells, threads, ..

2. Incremental development Definition Piece−by−piece

1 piece = 1 week of programming/testing, for example Definition Preserve software in sound/working state Related life cycle models Main problem: Why study incremental development ? −quality preservation difficult to achieve because the software is constantly changing Related life cycle models Why study incremental development?

Incremental development ~ Incremental development appears suitable for open source projects

~ iterative development Incremental development is key property of Bazaar Model: " " but: in Boehm’s Spiral model, analysis and design Release early, release often => ”parallel debugging” is required in each iteration " community of user/developers (or just advanced users) Incremental development is related to ”flat” organization ~ evolutionary prototyping " long−term planning difficult without sales revenues " but: purpose of e.p. is to obtain user feedback on design " write and read permissions (check−in, check out) " when to integrate: hundreds of decision makers ~ software development as software maintenance " document−based progress assessment impossible (aka evolutionary system development) Incremental development challenges classical software engineering " bugfixes and new featueres " design then code vs. code−n−fix are merely different kind of changes " systematic testing vs. ad hoc debugging

3. More FreeBSD background The FreeBSD product & service

Software is Operating system kernel + ports The FreeBSD product & service. " kernel: state of the art (eg. networking, virtual memory) " 4000+ pieces of third party software " 50+ Java−related ports (compilers, JVMs, JIT−compilers) The FreeBSD release tree Release 4.2 contains 75 diverse items listed in release notes The FreeBSD organization " adaptive changes: new drivers, ports updates, .. " corrective changes: 10 security fixes, .. The required design effort " perfective changes: improved TCP/IP performance, .. Several security advisories each month " victim program, problem, impact, workaround, fix " five versions of Bash shell fix distributed (3 Intel x86, 2 Compaq Alpha) The FreeBSD release tree The FreeBSD organization 3.0 Major release Core team (approx. 10), committers (200+), contributors (1200+) Minor 4.1 4.0 Snapshot Absolutely necessary to have well−defined and well−understood processes because development branch is ”melting pot” 4.2 Core team elected by ”committers” for 2 years " ”being able to work together is our greatest asset” " assigns managers of documentation, PR, web, mailing lists, .. " manages access to repository Major releases along main branch every 18 months (approx.) " development decisions

4−5 minor releases each year as improvements on latest major release Basic organizational ”unit” is maintainer " sources are maintained by formal or ”de facto” maintainer Distribution " maintainers are responsible for all changes not just corrective " major/minor releases on CD " " snapshots and major/minor releases always downloadable committers do corrective, perfective, etc. work interchangeably

The required design effort 4. FreeBSD’s "Life cycle for changes"

"Code, then fix" works for Linux (or FreeBSD) because: code " review pre−commit "By the time Linux came around, requirements and architecture development test parallel release production defects had already been flushed out during the development of debugging many previousgenerations of Unix" release (Steve McConnell, IEEE Software Magazine) I postulate a six phase model Argument may apply even more to FreeBSD " applies to each individual change (bugfix, small feature, part of large feat.) " FreeBSD goes back 20+ years " consistent with but not immediately apparent in Handbook, guidelines, etc. " Software that has been widely studied / taught " model consists of phases + key points for each phase

New requirements: Internet standards and processor technology I believe model captures "essense" of FreeBSD’s development process " why it works " some requirements are very demanding " model is explaining, captures mechanism " symmetric multi−processing " not just descriptive " but most work on FreeBSD is maintenaince−type of work Method (i) Code

code review pre−commit development test parallel release production debugging release Method Code guidelines exist " webbased survey involving 72 project participants Visibility of code is important: " quantitative + qualitative " 85% encouraged to improve skills Questions designed to verify/falsify (partially) because of visibility (Q 26) " Eric S. Raymond’s Bazaar model " "Embarrassment is a powerful thing" " Classical life cycle model " Hypotheses about learning

(ii) Review (iii) Pre−commit test

code code review pre−commit review pre−commit development development test parallel test parallel release production release production debugging debugging release release

Reviews strongly suggested but not absolutely required The implied responsibility of the committer to integrate changes is a learning incentive Committers frequently seek and receive code review " " 85% distributed code for review last 3 months 82% had improved their technical skills by debugging build failures Main obstacle to more frequent reviews on design is lack of feedback " ""Getting design feedback is like pulling teeth" Pre−commit test is really integration test ! (iv) Development release (v) Parallel debugging

code code review pre−commit review pre−commit development development test parallel test parallel release production release production debugging debugging release release

Top priority " "If you know of any bugfixes which have been successfully Commit authority is highly motivating applied to −CURRENT but have not been merged into − STABLE after a decent interval, send the committer a polite " 81% encouraged a lot by ability to integrate directly reminder" (Handbook)

" "I don’t feel I’m at the whim of a single person; I can Produces significant stream of PRs and other feedback develop/commit under my own authority, and possibly be overridden by a general consensus" But mostly on simple problems " "The bug reports we’ve gotten have been of limited use .. " "There is a tremendous satisfaction to the ’see bug, fix bug, the subtle problems are too ’unusual’ for most of the other see bug fix get incorporated so that the fix helps others’ cycle" developers to diagnose" (SMP project manager)

(vi) Production release 5. Case study of a major new feature: SMP (Symmetric Multi−Processing) Utilize cheap Intel multiprocessor architecture: 1 cabinet, 1 RAM, X CPUs code review pre−commit development test parallel release production Operating system kernel (large oval) must be re−entrant debugging release " raises many new design problems " requires fine−grained locking of kernel ressources (small ovals)

One of few deadlines. Evolutionary approach to keep kernel working (on all architectures) " 95% had no task deadline using temporary ”giant lock” (rectangle) " release 5.0 delayed 1+ year, oops Decision to use development branch (avoid unmanageable, late integration) Prior to major/minor release, authority " unstable development branch is delegated to a single individual " ongoing integration with other work is extremely difficult " committers retain commit privileges " "having people touch the same code is somewhat disruptive" " "It was a really boring month, but the 4.x release is probably the best we’ve had" 6. Conclusion

Incremental approach successful, model claims reason is: " motivates developers (ability to integrate changes directly) " bugfixing is prioritized (maintainer’s mixed responsibility) " encourages learning (via visibility and mandatory integration work) " responsibility, authority => motivation & learning

SMP subproject also used classical approach: " tasks were broken down and organized into a planned sequence " changes were described in separate design documents

SMP subproject faced severe problems: " decreased performance, partly because of extra locking " de−stabilized development branch annoying and required extra work " feedback from outside subproject was not valuable " therefore, incremental life cycle has its limitations