The Cathedral and the Bazaar
by Eric S. Raymond D ate : 1997=09=1522 : 48 : 58
I anatomize a successful free-software project, fetchmail, that was run as a delib erate test of some surprising
theories ab out software engineering suggested by the history of Linux. I discuss these theories in terms of two
fundamentally di erent development styles, the \cathedral" mo del of FSF and its imitators versus the \bazaar"
mo del of the Linux world. I show that these mo dels derive from opp osing assumptions ab out the nature of the
software-debugging task. I then make a sustained argument from the Linux exp erience for the prop osition that
\Given enough eyeballs, all bugs are shallow", suggest pro ductive analogies with other self-correcting systems of
sel sh agents, and conclude with some exploration of the implications of this insight for the future of software.
Contents
1 The Cathedral and the Bazaar 2
2 The Mail Must Get Through 2
3 The Imp ortance of Having Users 5
4 Release Early, Release Often 5
5 When Is A Rose Not A Rose? 8
6 Pop client b ecomes Fetchmail 9
7 Fetchmail Grows Up 11
8 AFew More Lessons From Fetchmail 12
9 Necessary Preconditions for the Bazaar Style 13
10 The So cial Context of Free Software 14
11 Acknowledgements 17
12 For Further Reading 17
13 Version Id : cathedr al paper:sg ml ; v 1:241997=09=1522 : 48 : 58esr E xpesr 18
1. The Cathedral and the Bazaar 2
1 The Cathedral and the Bazaar
Linux is subversive. Who would have thoughteven veyears ago that a world-class op erating system could
coalesce as if by magic out of part-time hacking by several thousand develop ers scattered all over the
planet, connected only by the tenuous strands of the Internet?
Certainly not I. By the time Linux swam onto my radar screen in early 1993, I had already b een involved
in Unix and free-software development for ten years. Iwas one of the rst GNU contributors in the
mid-1980s. I had released a go o d deal of free software onto the net, developing or co-developing several
programs nethack, Emacs VC and GUD mo des, xlife, and others that are still in wide use to day. I
thought I knew howitwas done.
Linux overturned much of what I thought I knew. I had b een preaching the Unix gosp el of small to ols,
rapid prototyping and evolutionary programming for years. But I also b elieved there was a certain critical
complexityabove which a more centralized, a priori approachwas required. I b elieved that the most
imp ortant software op erating systems and really large to ols like Emacs needed to b e built like cathedrals,
carefully crafted by individual wizards or small bands of mages working in splendid isolation, with no b eta
to b e released b efore its time.
Linus Torvalds's style of development - release early and often, delegate everything you can, b e op en to the
p oint of promiscuity - came as a surprise. No quiet, reverent cathedral-building here { rather, the Linux
community seemed to resemble a great babbling bazaar of di ering agendas and approaches aptly
symb olized by the Linux archive sites, who'd take submissions from anyone out of which a coherent and
stable system could seemingly emerge only by a succession of miracles.
The fact that this bazaar style seemed to work, and work well, came as a distinct sho ck. As I learned my
way around, I worked hard not just at individual pro jects, but also at trying to understand why the Linux
world not only didn't y apart in confusion but seemed to go from strength to strength at a sp eed barely
imaginable to cathedral-builders.
By mid-1996 I thoughtIwas b eginning to understand. Chance handed me a p erfect way to test my theory,
in the form of a free-software pro ject which I could consciously try to run in the bazaar style. So I did {
and it was a signi cant success.
In the rest of this article, I'll tell the story of that pro ject, and I'll use it to prop ose some aphorisms ab out
e ective free-software development. Not all of these are things I rst learned in the Linux world, but we'll
see how the Linux world gives them particular p oint. If I'm correct, they'll help you understand exactly
what it is that makes the Linux community such a fountain of go o d software { and help you b ecome more
pro ductiveyourself.
2 The Mail Must Get Through
Since 1993 I'd b een running the technical side of a small free-access ISP called Chester CountyInterLink
CCIL in West Chester, Pennsylvania I co-founded CCIL and wrote our unique multiuser BBS software {
you can check it out by telnetting to locke.ccil.org
three thousand users on nineteen lines. The job allowed me 24-hour-a-day access to the net through CCIL's 56K line { in fact, it practically demanded it!
2. The Mail Must Get Through 3
Accordingly, I had gotten quite used to instantInternet email. For complicated reasons, it was hard to get
SLIP to work b etween my home machine snark.thyrsus.com and CCIL. When I nally succeeded, I found
having to p erio dically telnet to lo cketocheckmy mail annoying. What I wanted was for my mail to b e
delivered on snark so that bi 1 would notify me when it arrived.
Simple sendmail forwarding wouldn't work, b ecause snark isn't always on the net and do esn't have a static
IP address. What I needed was a a program that would reach out over my SLIP connection and pull across
my mail to b e delivered lo cally. I knew such things existed, and that most of them used a simple
application proto col called POP Post Oce Proto col . And sure enough, there was already a POP3 server
included with lo cke's BSD/OS op erating system.
I needed a POP3 client. So I went out on the net and found one. Actually, I found three or four. I used
p op-p erl for a while, but it was missing what seemed an obvious feature, the ability to hack the addresses
on fetched mail so replies would work prop erly.
The problem was this: supp ose someone named `jo e' on lo cke sent me mail. If I fetched the mail to snark
and then tried to reply to it, my mailer would cheerfully try to ship it to a nonexistent `jo e' on snark.
Hand-editing reply addresses to tack on `@ccil.org' quickly got to b e a serious pain.
This was clearly something the computer ought to b e doing for me. In fact, according to RFC1123 section
5.2.18, sendmail ought to b e doing it. But none of the existing POP clients knew how! And this brings us
to the rst lesson:
1. Every good work of software starts by scratching a developer's personal itch.
Perhaps this should have b een obvious it's long b een proverbial that \Necessity is the mother of
invention" but to o often software develop ers sp end their days grinding away for pay at programs they
neither need nor love. But not in the Linux world { whichmay explain why the average quality of software
originated in the Linux community is so high.
So, did I immediately launchinto a furious whirl of co ding up a brand-new POP3 client to comp ete with
the existing ones? Not on your life! Ilooked carefully at the POP utilities I had in hand, asking myself
\which one is closest to what I want?". Because
2. Goodprogrammers know what to write. Great ones know what to rewrite and reuse .
While I don't claim to b e a great programmer, I try to imitate one. An imp ortant trait of the great ones is
constructive laziness. They know that you get an A not for e ort but for results, and that it's almost
always easier to start from a go o d partial solution than from nothing at all.
Linus, for example, didn't actually try to write Linux from scratch. Instead, he started by reusing co de and
ideas from Minix, a tiny Unix-like OS for 386 machines. Eventually all the Minix co de wentawayorwas
completely rewritten { but while it was there, it provided sca olding for the infant that would eventually
b ecome Linux.
In the same spirit, I went lo oking for an existing POP utility that was reasonably well co ded, to use as a
development base.
The source-sharing tradition of the Unix world has always b een friendly to co de reuse this is why the
GNU pro ject chose Unix as a base OS, in spite of serious reservations ab out the OS itself . The Linux
world has taken this tradition nearly to its technological limit; it has terabytes of op en sources generally
available. So sp ending time lo oking for some else's almost-go o d-enough is more likely to giveyou go o d
2. The Mail Must Get Through 4
results in the Linux world than anywhere else.
And it did for me. With those I'd found earlier, my second search made up a total of nine candidates {
fetchp op, PopTart, get-mail, gwp op, pimp, p op-p erl, p op c, p opmail and up op. The one I rst settled on
was `fetchp op' by Seung-Hong Oh. I put my header-rewrite feature in it, and made various other
improvements which the author accepted into his 1.9 release.
A few weeks later, though, I stumbled across the co de for `p op client' by Carl Harris, and found I had a
problem. Though fetchp op had some go o d original ideas in it such as its daemon mo de , it could only
handle POP3 and was rather amateurishly co ded Seung-Hong was a bright but inexp erienced programmer,
and b oth traits showed . Carl's co de was b etter, quite professional and solid, but his program lacked
several imp ortant and rather tricky-to-implement fetchp op features including those I'd co ded myself .
Stay or switch? If I switched, I'd b e throwing away the co ding I'd already done in exchange for a b etter
development base.
A practical motive to switchwas the presence of multiple-proto col supp ort. POP3 is the most commonly
used of the p ost-oce server proto cols, but not the only one. Fetchp op and the other comp etition didn't
do POP2, RPOP, or APOP, and I was already having vague thoughts of p erhaps adding IMAP Internet
Message Access Proto col, the most recently designed and most p owerful p ost-oce proto col just for fun.
But I had a more theoretical reason to think switching might b e as go o d an idea as well, something I
learned long b efore Linux.
3. \Plan to throw one away; you wil l, anyhow." FredBrooks, \The Mythical Man-Month", Chapter 11
Or, to put it another way,you often don't really understand the problem until after the rst time you
implement a solution. The second time, mayb e you know enough to do it right. So if you want to get it
right, b e ready to start over at least once.
Well I told myself the changes to fetchp op had b een my rst try. So I switched.
After I sentmy rst set of p op client patches to Carl Harris on 25 June 1996, I found out that he had
basically lost interest in p op client some time b efore. The co de was a bit dusty, with minor bugs hanging
out. I had manychanges to make, and we quickly agreed that the logical thing for me to do was takeover
the program.
Without my actually noticing, the pro ject had escalated. No longer was I just contemplating minor patches
to an existing POP client. I to ok on maintaining an entire one, and there were ideas bubbling in my head
that I knew would probably lead to ma jor changes.
In a software culture that encourages co de-sharing, this is a natural way for a pro ject to evolve. Iwas
acting out this:
4. If you have the right attitude, interesting problems wil l nd you.
But Carl Harris's attitude was even more imp ortant. He understo o d that
5. When you lose interest in a program, your last duty to it is to hand it o to a competent successor.
Without ever having to discuss it, Carl and I knew we had a common goal of having the b est solution out
there. The only question for either of us was whether I could establish that I was a safe pair of hands.
Once I did that, he acted with grace and dispatch. I hop e I will act as well when it comes my turn.
3. The Imp ortance of Having Users 5
3 The Imp ortance of Having Users
And so I inherited p op client. Just as imp ortantly, I inherited p op client's user base. Users are wonderful
things to have, and not just b ecause they demonstrate that you're serving a need, that you've done
something right. Prop erly cultivated, they can b ecome co-develop ers.
Another strength of the Unix tradition, and again one that Linux pushes to a happy extreme, is that a lot
of users are hackers to o { and b ecause source co de is available, they can b e e ective hackers. This can b e
tremendously useful for shortening debugging time. Given a bit of encouragement, your users will diagnose
problems, suggest xes, and help improve the co de far more quickly than you could unaided.
6. Treating your users as co-developers is your least-hassle route to rapid code improvement and e ective
debugging.
The p ower of this e ect is easy to underestimate. In fact, prettywell all of us in the free-software world
drastically underestimated howwell it would scale up with numb er of users and against system complexity,
until Linus showed us di erently.
In fact, I think Linus's cleverest and most consequential hackwas not the construction of the Linux kernel
itself, but rather his invention of the Linux development mo del. When I expressed this opinion in his
presence once, he smiled and quietly rep eated something he has often said: \I'm basically a very lazy
p erson who likes to get credit for things other p eople actually do." Lazy likeafox. Or, as Rob ert Heinlein
mighthave said, to o lazy to fail.
In retrosp ect, one precedent for the metho ds and success of Linux can b e seen in the development of the
GNU Emacs Lisp library and Lisp co de archives. In contrast to the cathedral-building style of the Emacs
C core and most other FSF to ols, the evolution of the Lisp co de p o ol was uid and very user-driven. Ideas
and prototyp e mo des were often rewritten three or four times b efore reaching a stable nal form. And
lo osely-coupled collab orations enabled by the Internet, a la Linux, were frequent.
Indeed, myown most successful single hack previous to fetchmail was probably Emacs VC mo de, a
Linux-like collab oration by email with three other p eople, only one of whom Richard Stallman I have met
to this day. It was a front-end for SCCS, RCS and later CVS from within Emacs that o ered \one-touch"
version control op erations. It evolved from a tiny, crude sccs.el mo de someb o dy else had written. And the
developmentofVC succeeded b ecause, unlike Emacs itself, Emacs Lisp co de could go through
release/test/improve generations very quickly.
One unexp ected side-e ect of FSF's p olicy of trying to legally bind co de into the GPL is that it b ecomes
pro cedurally harder for FSF to use the bazaar mo de, since they b elieve they must get a copyright
assignment for every individual contribution of more than twenty lines in order to immunize GPLed co de
from challenge under copyrightlaw. Users of the BSD and MIT X Consortium licenses don't have this
problem, since they're not trying to reserve rights that anyone mighthave an incentivetochallenge.
4 Release Early, Release Often
Early and frequent releases are a critical part of the Linux development mo del. Most develop ers including
me used to b elieve this was bad p olicy for larger than trivial pro jects, b ecause early versions are almost by
4. Release Early, Release Often 6
de nition buggy versions and you don't wanttowear out the patience of your users.
This b elief reinforced the general commitment to a cathedral-building style of development. If the
overriding ob jectivewas for users to see as few bugs as p ossible, why then you'd only release one every six
months or less often and work like a dog on debugging b etween releases. The Emacs C core was
develop ed this way. The Lisp library, in e ect, was not { b ecause there were active Lisp archives outside
the FSF's control, where you could go to nd new and developmentcodeversions indep endently of Emacs's
release cycle.
The most imp ortant of these, the Ohio State elisp archive, anticipated the spirit and many of the features
of to day's big Linux archives. But few of us really thoughtvery hard ab out what wewere doing, or ab out
what the very existence of that archive suggested ab out problems in FSF's cathedral-building development
mo del. I made one serious attempt around 1992 to get a lot of the Ohio co de formally merged into the
ocial Emacs Lisp library. I ran into p olitical trouble and was largely unsuccessful.
But byayear later, as Linux b ecame widely visible, it was clear that something di erent and much
healthier was going on there. Linus's op en development p olicy was the very opp osite of cathedral-building.
The sunsite and tsx-11 archives were burgeoning, multiple distributions were b eing oated. And all of this
was driven by an unheard-of frequency of core system releases.
Linus was treating his users as co-develop ers in the most e ective p ossible way:
7. Release early. Release often. And listen to your customers.
Linus's innovation wasn't so much in doing this something like it had b een Unix-world tradition for a long
time , but in scaling it up to a level of intensity that matched the complexity of what he was developing.
In those early times it wasn't unknown for him to release a new kernel more than once a day! And,
b ecause he cultivated his base of co-develop ers and leveraged the Internet for collab oration harder than
anyone else, it worked.
But how did it work? And was it something I could duplicate, or did it rely on some unique genius of
Linus's?
I didn't think so. Granted, Linus is a damn ne hacker how many of us could engineer an entire
pro duction-quality op erating system kernel? . But Linux didn't representanyawesome conceptual leap
forward. Linus is not or at least, not yet an innovative genius of design in the way that, say, Richard
Stallman or James Gosling are. Rather, Linus seems to me to b e a genius of engineering, with a sixth sense
for avoiding bugs and development dead-ends and a true knack for nding the minimum-e ort path from
p ointAtopoint B. Indeed, the whole design of Linux breathes this quality and mirrors Linus's essentially
conservative and simplifying design approach.
So, if rapid releases and leveraging the Internet medium to the hilt were not accidents but integral parts of
Linus's engineering-genius insightinto the minimum-e ort path, what was he maximizing? What was he
cranking out of the machinery?
Put that way, the question answers itself. Linus was keeping his hacker/users constantly stimulated and
rewarded { stimulated by the prosp ect of having an ego-satisfying piece of the action, rewarded by the
sight of constant even daily improvement in their work.
Linus was directly aiming to maximize the numb er of p erson-hours thrown at debugging and development,
even at the p ossible cost of instability in the co de and user-base burnout if any serious bug proved
4. Release Early, Release Often 7
intractable. Linus was b ehaving as though he b elieved something like this:
8. Given a large enough beta-tester and co-developer base, almost every problem wil l be characterized
quickly and the x obvious to someone.
Or, less formally, \Given enough eyeballs, all bugs are shallow." I dub this: \Linus's Law".
My original formulation was that every problem \will b e transparent to someb o dy". Linus demurred that
the p erson who understands and xes the problem is not necessarily or even usually the p erson who rst
characterizes it. \Someb o dy nds the problem," he says, \and someb o dy else understands it. And I'll go
on record as saying that nding it is the bigger challenge." But the p oint is that b oth things tend to
happ en quickly.
Here, I think, is the core di erence underlying the cathedral-builder and bazaar styles. In the
cathedral-builder view of programming, bugs and development problems are tricky, insidious, deep
phenomena. It takes months of scrutinyby a dedicated few to develop con dence that you've winkled them
all out. Thus the long release intervals, and the inevitable disapp ointment when long-awaited releases are
not p erfect.
In the bazaar view, on the other hand, you assume that bugs are generally shallow phenomena { or, at
least, that they turn shallow pretty quick when exp osed to a thousand eager co-develop ers p ounding on
every single new release. Accordingly you release often in order to get more corrections, and as a b ene cial
side e ect you have less to lose if an o ccasional b otch gets out the do or.
And that's it. That's enough. If \Linus's Law" is false, then any system as complex as the Linux kernel,
b eing hacked over byasmany hands as the Linux kernel, should at some p ointhave collapsed under the
weight of unforseen bad interactions and undiscovered \deep" bugs. If it's true, on the other hand, it is
sucient to explain Linux's relative lack of bugginess.
And mayb e it shouldn't have b een such a surprise, at that. So ciologists years ago discovered that the
averaged opinion of a mass of equally exp ert or equally ignorant observers is quite a bit more reliable a
predictor than that of a single randomly-chosen one of the observers. They called this the \Delphi e ect".
It app ears that what Linus has shown is that this applies even to debugging an op erating system { that the
Delphi e ect can tame development complexityeven at the complexity level of an OS kernel.
I am indebted to Je Dutky
as \Debugging is parallelizable". Je observes that although debugging requires debuggers to communicate
with some co ordinating develop er, it do esn't require signi cant co ordination b etween debuggers. Thus it
do esn't fall prey to the same quadratic complexity and management costs that make adding develop ers
problematic.
In practice, the theoretical loss of eciency due to duplication of work by debuggers almost never seems to
b e an issue in the Linux world. One e ect of a \release early and often p olicy" is to minimize such
duplication by propagating fed-back xes quickly.
Bro oks even made an o -hand observation related to Je 's: \The total cost of maintaining a widely used
program is typically 40 p ercent or more of the cost of developing it. Surprisingly this cost is strongly
a ected by the numb er of users. More users nd more bugs ." my emphasis .
More users nd more bugs b ecause adding more users adds more di erentways of stressing the program.
This e ect is ampli ed when the users are co-develop ers. Each one approaches the task of bug
5. When Is A Rose Not A Rose? 8
characterization with a slightly di erent p erceptual set and analytical to olkit, a di erent angle on the
problem. The \Delphi e ect" seems to work precisely b ecause of this variation. In the sp eci c context of
debugging, the variation also tends to reduce duplication of e ort.
So adding more b eta-testers may not reduce the complexity of the current \deep est" bug from the
developer's P.O.V., but it increases the probability that someone's to olkit will b e matched to the problem
in suchaway that the bug is shallow to that person .
Linus copp ers his b ets, to o. In case there are serious bugs, Linux kernel version are numb ered in sucha
way that p otential users can makea choice either to run the last version designated \stable" or to ride the
cutting edge and risk bugs in order to get new features. This tactic is not yet formally imitated by most
Linux hackers, but p erhaps it should b e; the fact that either choice are available makes b oth more
attractive.
5 When Is A Rose Not A Rose?
Having studied Linus's b ehavior and formed a theory ab out whyitwas successful, I made a conscious
decision to test this theory on my new admittedly much less complex and ambitious pro ject.
But the rst thing I did was reorganize and simplify p op client a lot. Carl Harris's implementation was very
sound, but exhibited a kind of unnecessary complexity common to many C programmers. He treated the
co de as central and the data structures as supp ort for the co de. As a result, the co de was b eautiful but the
data structure design ad-ho c and rather ugly at least by the high standards of this old LISP hacker .
I had another purp ose for rewriting b esides improving the co de and the data structure design, however.
That was to evolveitinto something I understo o d completely. It's no fun to b e resp onsible for xing bugs
in a program you don't understand.
For the rst month or so, then, I was simply following out the implications of Carl's basic design. The rst
serious change I made was to add IMAP supp ort. I did this by reorganizing the proto col machines into a
generic driver and three metho d tables for POP2, POP3, and IMAP . This and the previous changes
illustrate a general principle that's go o d for programmers to keep in mind, esp ecially in languages likeC
that don't naturally do dynamic typing:
9. Smart data structures and dumb code works a lot better than the other way around.
Fred Bro oks, Chapter 11 again: \Showmeyour code and conceal your datastr uctur es, and I shall continue
to b e mysti ed. Showmeyour datastr uctur es, and I won't usually need your code; it'll b e obvious."
Actually, he said \ owcharts" and \tables". But allowing for thirtyyears of terminological/cultural shift,
it's almost the same p oint.
At this p oint early Septemb er 1996, ab out six weeks from zero I started thinking that a name change
might b e in order { after all, it wasn't just a POP clientany more. But I hesitated, b ecause there was as
yet nothing genuinely new in the design. My version of p op client had yet to develop an identity of its own.
That changed, radically, when fetchmail learned how to forward fetched mail to the SMTP p ort. I'll get to
that in a moment. But rst: I said ab ove that I'd decided to use this pro ject to test my theory ab out what
Linus Torvalds had done right. How you maywell ask did I do that? In these ways:
6. Pop client b ecomes Fetchmail 9
1. I released early and often almost never less often than every ten days; during p erio ds of intense
development, once a day .
2. I grew my b eta list by adding to it everyone who contacted me ab out fetchmail.
3. I sentchatty announcements to the b eta list whenever I released, encouraging p eople to participate.
4. And I listened to my b eta testers, p olling them ab out design decisions and stroking them whenever
they sent in patches and feedback.
The payo from these simple measures was immediate. From the b eginning of the pro ject, I got bug
rep orts of a quality most develop ers would kill for, often with go o d xes attached. I got thoughtful
criticism, I got fan mail, I got intelligent feature suggestions. Which leads to:
10. If you treat your beta-testers as if they're your most valuable resource, they wil l respond by becoming
your most valuable resource.
One interesting measure of fetchmail's success is the sheer size of the pro ject b eta list, fetchmail-friends. At
time of writing it has 249 memb ers and is adding two or three a week.
Actually, as I revise in late May 1997 the list is b eginning to lose memb ers for an interesting reason.
Several p eople have asked me to unsubscrib e them b ecause fetchmail is working so well for them that they
no longer need to see the list trac! Perhaps this is part of the normal life-cycle of a mature bazaar-style
pro ject.
6 Pop client b ecomes Fetchmail
The real turning p oint in the pro ject was when Harry Ho chheiser sent me his scratch co de for forwarding
mail to the client machine's SMTP p ort. I realized almost immediately that a reliable implementation of
this feature would make all the other delivery mo des next to obsolete.
For manyweeks I had b een tweaking fetchmail rather incrementally while feeling like the interface design
was serviceable but grubby { inelegant and with to o many exiguous options hanging out all over. The
options to dump fetched mail to a mailb ox le or standard output particularly b othered me, but I couldn't
gure out why.
What I saw when I thought ab out SMTP forwarding was that p op client had b een trying to do to o many
things. It had b een designed to b e b oth a mail transp ort agent MTA and a lo cal delivery agent MDA .
With SMTP forwarding, it could get out of the MDA business and b e a pure MTA, handing o mail to
other programs for lo cal delivery just as sendmail do es.
Why mess with all the complexity of con guring a mail delivery agent or setting up lo ck-and-app end on a
mailb ox when p ort 25 is almost guaranteed to b e there on any platform with TCP/IP supp ort in the rst
place? Esp ecially when this means retrieved mail is guaranteed to lo ok like normal sender-initiated SMTP
mail, which is really what wewantanyway.
There are several lessons here. First, this SMTP-forwarding idea was the biggest single payo I got from
consciously trying to emulate Linus's metho ds. A user gave me this terri c idea { all I had to do was understand the implications.
6. Pop client b ecomes Fetchmail 10
11. The next best thing to having good ideas is recognizing good ideas from your users. Sometimes the latter
is better.
Interestingly enough, you will quickly nd that if you are completely and self-deprecatingly truthful ab out
howmuchyou owe other p eople, the world at large will treat you likeyou did every bit of the invention
yourself and are just b eing b ecomingly mo dest ab out your innate genius. We can all see howwell this
worked for Linus!
And after a very few weeks of running the pro ject in the same spirit, I b egan to get similar praise not just
from my users but from other p eople to whom the word leaked out. I stashed away some of that email; I'll
lo ok at it again sometime if I ever start wondering whether my life has b een worthwhile :- .
But there are two more fundamental, non-p olitical lessons here that are general to all kinds of design.
12. Often, the most striking and innovative solutions come from realizing that your concept of the problem
was wrong.
I had b een trying to solve the wrong problem by continuing to develop p op client as a combined
MTA/MDA with all kinds of funky lo cal delivery mo des. Fetchmail's design needed to b e rethought from
the ground up as a pure MTA, a part of the normal SMTP-sp eaking Internet mail path.
When you hit a wall in development { when you nd yourself hard put to think past the next patch { it's
often time to ask not whether you've got the right answer, but whether you're asking the right question.
Perhaps the problem needs to b e reframed.
Well, I had reframed my problem. Clearly, the right thing to do was 1 hack SMTP forwarding supp ort
into the generic driver, 2 make it the default mo de, and 3 eventually throw out all the other delivery
mo des, esp ecially the deliver-to- le and deliver-to-standard-output options.
I hesitated over step 3 for some time, fearing to upset long-time p op client users dep endent on the alternate
delivery mechanisms. In theory, they could immediately switch to .forward les or their non-sendmail
equivalents to get the same e ects. In practice the transition mighthave b een messy.
But when I did it, the b ene ts proved huge. The cruftiest parts of the driver co de vanished. Con guration
got radically simpler { no more grovelling around for the system MDA and user's mailb ox, no more worries
ab out whether the underlying OS supp orts le lo cking.
Also, the only way to lose mail vanished. If you sp eci ed delivery to a le and the disk got full, your mail
got lost. This can't happ en with SMTP forwarding b ecause your SMTP listener won't return OK unless
the message can b e delivered or at least sp o oled for later delivery.
Also, p erformance improved though not so you'd notice it in a single run . Another not insigni cant
b ene t of this change was that the manual page got a lot simpler.
Later, I had to bring delivery via a user-sp eci ed lo cal MDA back in order to allow handling of some
obscure situations involving dynamic SLIP. But I found a much simpler way to do it.
The moral? Don't hesitate to throwaway sup erannuated features when you can do it without loss of
e ectiveness. Antoine de Saint-Exup ery who was an aviator and aircraft designer when he wasn't b eing
the author of classic children's b o oks said:
13. \Perfection in design is achieved not when there is nothing more to add, but rather when thereis
nothing more to take away."
7. Fetchmail Grows Up 11
When your co de is getting b oth b etter and simpler, that is when you know it's right. And in the pro cess,
the fetchmail design acquired an identity of its own, di erent from the ancestral p op client.
It was time for the name change. The new design lo oked much more like a dual of sendmail than the old
p op client had; b oth are MTAs, but where sendmail pushes then delivers, the new p op client pulls then
delivers. So, two months o the blo cks, I renamed it fetchmail.
7 Fetchmail Grows Up
There I was with a neat and innovative design, co de that I knew worked well b ecause I used it every day,
and a burgeoning b eta list. It gradually dawned on me that I was no longer engaged in a trivial p ersonal
hack that might happ en to b e useful to few other p eople. I had my hands on a program every hacker with
a Unix b ox and a SLIP/PPP mail connection really needs.
With the SMTP forwarding feature, it pulled far enough in front of the comp etition to p otentially b ecome
a \category killer", one of those classic programs that lls its niche so comp etently that the alternatives are
not just discarded but almost forgotten.
I think you can't really aim or plan for a result like this. You have to get pulled into it by design ideas so
powerful that afterward the results just seem inevitable, natural, even foreordained. The only way to try
for ideas like that is byhaving lots of ideas { or byhaving the engineering judgmenttotake other p eoples'
go o d ideas b eyond where the originators thought they could go.
Andrew Tanenbaum had the original idea to build a simple native Unix for the 386, for use as a teaching
to ol. Linus Torvalds pushed the Minix concept further than Andrew probably thought it could go { and it
grew into something wonderful. In the same way though on a smaller scale , I to ok some ideas by Carl
Harris and Harry Ho chheiser and pushed them hard. Neither of us was `original' in the romantic way
p eople think is genius. But then, most science and engineering and software development isn't done by
original genius, hacker mythology to the contrary.
The results were pretty heady stu all the same { in fact, just the kind of success every hacker lives for!
And they meantI would have to set my standards even higher. To make fetchmail as go o d as I nowsawit
could b e, I'd have to write not just for myown needs, but also include and supp ort features necessary to
others but outside my orbit. And do that while keeping the program simple and robust.
The rst and overwhelmingly most imp ortant feature I wrote after realizing this was multidrop supp ort {
the ability to fetch mail from mailb oxes that had accumulated all mail for a group of users, and then route
each piece of mail to its individual recipients.
I decided to add the multidrop supp ort partly b ecause some users were clamoring for it, but mostly
b ecause I thoughtitwould shake bugs out of the single-drop co de by forcing me to deal with addressing in
full generality. And so it proved. Getting RFC822 parsing right to ok me a remarkably long time, not
b ecause any individual piece of it is hard but b ecause it involved a pile of interdep endent and fussy details.
But multidrop addressing turned out to b e an excellent design decision as well. Here's how I knew:
14. Any tool should be useful in the expected way, but a *great* tool lends itself to uses you never expected.
The unexp ected use for multi-drop fetchmail is to run mailing lists with the list kept, and alias expansion
8. AFew More Lessons From Fetchmail 12
done, on the client side of the SLIP/PPP connection. This means someone running a p ersonal machine
through an ISP account can manage a mailing list without continuing access to the ISP's alias les.
Another imp ortantchange demanded bymy b eta testers was supp ort for 8-bit MIME op eration. This was
pretty easy to do, b ecause I had b een careful to keep the co de 8-bit clean. Not b ecause I anticipated the
demand for this feature, but rather in ob edience to another rule:
15. When writing gateway software of any kind, take pains to disturb the data stream as little as possible {
and *never* throw away information unless the recipient forces you to!
Had I not ob eyed this rule, 8-bit MIME supp ort would have b een dicult and buggy. As it was, all I had
to do is read RFC 1652 and add a trivial bit of header-generation logic.
Some Europ ean users bugged me into adding an option to limit the numb er of messages retrieved p er
session so they can control costs from their exp ensive phone networks . I resisted this for a long time, and
I'm still not entirely happy ab out it. But if you're writing for the world, you have to listen to your
customers { this do esn't change just b ecause they're not paying you in money.
8 A Few More Lessons From Fetchmail
Before wegoback to general software-engineering issues, there are a couple more sp eci c lessons from the
fetchmail exp erience to p onder.
The rc le syntax includes optional `noise' keywords that are entirely ignored by the parser. The
English-like syntax they allow is considerably more readable than the traditional terse keyword-value pairs
you get when you strip them all out.
These started out as a late-night exp eriment when I noticed howmuch the rc le declarations were
b eginning to resemble an imp erative minilanguage. This is also whyIchanged the original p op client
`server' keyword to `p oll' .
It seemed to me that trying to make that imp erative minilanguage more like English might make it easier
to use. Now, although I'm a convinced partisan of the \make it a language" scho ol of design as exempli ed
by Emacs and HTML and many database engines, I am not normally a big fan of \English-like" syntaxes.
Traditionally programmers have tended to favor control syntaxes that are very precise and compact and
have no redundancy at all. This is a cultural legacy from when computing resources were exp ensive, so
parsing stages had to b e as cheap and simple as p ossible. English, with ab out 50 redundancy,looked like
avery inappropriate mo del then.
This is not my reason for ghting shy of English-like syntaxes; I mention it here only to demolish it. With
cheap cycles and core, terseness should not b e an end in itself. Nowadays it's more imp ortant for a
language to b e convenient for humans than to b e cheap for the computer.
There are, however, go o d reasons to b e wary. One is the complexity cost of the parsing stage { you don't
want to raise that to the p oint where it's a signi cant source of bugs and user confusion in itself. Another
is that trying to make a language syntax English-like often demands that the \English" it sp eaks b e b ent
seriously out of shap e, so much so that the sup er cial resemblance to natural language is as confusing as a
traditional syntax would have b een. You see this in a lot of 4GLs and commercial database-query
9. Necessary Preconditions for the Bazaar Style 13
languages.
The fetchmail control syntax seems to avoid these problems b ecause the language domain is extremely
restricted. It's nowhere near a general-purp ose language; the things it says simply are not very
complicated, so there's little p otential for confusion in moving mentally b etween a tiny subset of English
and the actual control language. I think there may b e a wider lesson here:
16. When your language is nowherenear Turing-complete, syntactic sugar can be your friend.
Another lesson is ab out securityby obscurity. Some fetchmail users asked me to change the software to
store passwords encrypted in the rc le, so sno op ers wouldn't b e able to casually see them.
I didn't do it, b ecause this do esn't actually add protection. Anyone who's acquired p ermissions to read
your rc le will b e able to run fetchmail as you anyway { and if it's your password they're after, they'd b e
able to rip the necessary deco der out of the fetchmail co de itself to get it.
All .fetchmailrc password encryption would have done is give a false sense of security to p eople who don't
think very hard. The general rule here is:
17. Asecurity system is only as secure as its secret. Beware of pseudo-secrets.
9 Necessary Preconditions for the Bazaar Style
Early reviewers and test audiences for this pap er consistently raised questions ab out the preconditions for
successful bazaar-style development, including b oth the quali cations of the pro ject leader and the state of
co de at the time one go es public and starts to try to build a co-develop er community.
It's fairly clear that one cannot co de from the ground up in bazaar style. One can test, debug and improve
in bazaar style, but it would b e very hard to originate a pro ject in bazaar mo de. Linus didn't try it. I
didn't either. Your nascent develop er community needs to have something runnable and testable to play
with.
When you start community-building, what you need to b e able to presentisaplausible promise . Your
program do esn't havetowork particularly well. It can b e crude, buggy, incomplete, and p o orly
do cumented. What it must not fail to do is convince p otential co-develop ers that it can b e evolved into
something really neat in the foreseeable future.
Linux and fetchmail b oth went public with strong, attractive basic designs. Many p eople thinking ab out
the bazaar mo del as I have presented it have correctly considered this critical, then jump ed from it to the
conclusion that a high degree of design intuition and cleverness in the pro ject leader is indisp ensable.
But Linus got his design from Unix. I got mine initially from the ancestral p opmail though it would later
change a great deal, much more prop ortionately sp eaking than has Linux . So do es the leader/co ordinator
for a bazaar-style e ort really havetohave exceptional design talent, or can he get by on leveraging the
design talent of others?
I think it is not critical that the co ordinator b e able to originate designs of exceptional brilliance, but it is
absolutely critical that he/she b e able to recognize good design ideas from others .
Both the Linux and fetchmail pro jects show evidence of this. Linus, while not as previously discussed a
10. The So cial Context of Free Software 14
sp ectacularly original designer, has displayedapowerful knack for recognizing go o d design and integrating
it into the Linux kernel. AndIhave already describ ed how the single most p owerful design idea in
fetchmail SMTP forwarding came from someb o dy else.
Early audiences of this pap er complimented me by suggesting that I am prone to undervalue design
originality in bazaar pro jects b ecause I have a lot of it myself, and therefore take it for granted. There may
b e some truth to this; design as opp osed to co ding or debugging is certainly my strongest skill.
But the problem with b eing clever and original in software design is that it gets to b e a habit { you start
re exively making things cute and complicated when you should b e keeping them robust and simple. I
have had pro jects crash on me b ecause I made this mistake, but I managed not to with fetchmail.
So I b elieve the fetchmail pro ject succeeded partly b ecause I restrained my tendency to b e clever; this
argues at least against design originality b eing essential for successful bazaar pro jects. And consider
Linux. Supp ose Linus Torvalds had b een trying to pull o fundamental innovations in op erating system
design during the development; do es it seem at all likely that the resulting kernel would b e as stable and
successful as what wehave?
A certain base level of design and co ding skill is required, of course, but I exp ect almost anyb o dy seriously
thinking of launching a bazaar e ort will already b e ab ove that minimum. The free-software community's
internal market in reputation exerts subtle pressure on p eople not to launch development e orts they're
not comp etent to follow through on. So far this seems to haveworked prettywell.
There is another kind of skill not normally asso ciated with software development which I think is as
imp ortant as design cleverness to bazaar pro jects { and it may b e more imp ortant. A bazaar pro ject
co ordinator or leader must have go o d p eople and communications skills.
This should b e obvious. In order to build a development community,you need to attract p eople, interest
them in what you're doing, and keep them happy ab out the amountofwork they're doing. Technical sizzle
will go a long waytowards accomplishing this, but it's far from the whole story. The p ersonalityyou
pro ject matters, to o.
It is not a coincidence that Linus is a nice guy who makes p eople like him and want to help him. It's not a
coincidence that I'm an energetic extrovert who enjoys working a crowd and has some of the delivery and
instincts of a stand-up comic. To make the bazaar mo del work, it helps enormously if you have at least a
little skill at charming p eople.
10 The So cial Context of Free Software
It is truly written: the b est hacks start out as p ersonal solutions to the author's everyday problems, and
spread b ecause the problem turns out to b e typical for a large class of users. This takes us backtothe
matter of rule 1, restated in a p erhaps more useful way:
18. To solve an interesting problem, start by nding a problem that is interesting to you.
So it was with Carl Harris and the ancestral p op client, and so with me and fetchmail. But this has b een
understo o d for a long time. The interesting p oint, the p oint that the histories of Linux and fetchmail seem
to demand we fo cus on, is the next stage { the evolution of software in the presence of a large and active
community of users and co-develop ers.
10. The So cial Context of Free Software 15
In \The Mythical Man-Month", Fred Bro oks observed that programmer time is not fungible; adding
develop ers to a late software pro ject makes it later. He argued that the complexity and communication
costs of a pro ject rise with the square of the numb er of develop ers, while work done only rises linearly.
This claim has since b ecome known as \Bro ok's Law" and is widely regarded as a truism. But if Bro oks's
Lawwere the whole picture, Linux would b e imp ossible.
A few years later Gerald Weinb erg's classic \The Psychology Of Computer Programming" supplied what,
in hindsight, we can see as a vital correction to Bro oks. In his discussion of \egoless programming",
Weinb erg observed that in shops where develop ers are not territorial ab out their co de, and encourage other
p eople to lo ok for bugs and p otential improvements in it, improvement happ ens dramatically faster than
elsewhere.
Weinb erg's choice of terminology has p erhaps prevented his analysis from gaining the acceptance it
deserved { one has to smile at the thought of describing Internet hackers as \egoless". But I think his
argument lo oks more comp elling to day than ever.
The history of Unix should have prepared us for what we're learning from Linux and what I'veveri ed
exp erimentally on a smaller scale by delib erately copying Linus's metho ds . That is, that while co ding
remains an essentially solitary activity, the really great hacks come from harnessing the attention and
brainp ower of entire communities. The develop er who uses only his or her own brain in a closed pro ject is
going to fall b ehind the develop er who knows how to create an op en, evolutionary context in which
bug-sp otting and improvements get done byhundreds of p eople.
But the traditional Unix world was prevented from pushing this approach to the ultimate by several
factors. One was the legal contraints of various licenses, trade secrets, and commercial interests. Another
in hindsight was that the Internet wasn't yet go o d enough.
Before cheap Internet, there were some geographically compact communities where the culture encouraged
Weinb erg's \egoless" programming, and a develop er could easily attract a lot of skilled kibitzers and
co-develop ers. Bell Labs, the MIT AI Lab, UC Berkeley { these b ecame the home of innovations that are
legendary and still p otent.
Linux was the rst pro ject to make a conscious and successful e ort to use the entire world as its talent
p o ol. I don't think it's a coincidence that the gestation p erio d of Linux coincided with the birth of the
World Wide Web, and that Linux left its infancy during the same p erio d in 1993-1994 that saw the takeo
of the ISP industry and the explosion of mainstream interest in the Internet. Linus was the rst p erson
who learned how to playby the new rules that p ervasiveInternet made p ossible.
While cheap Internet was a necessary condition for the Linux mo del to evolve, I think it was not by itself a
sucient condition. Another vital factor was the development of a leadership style and set of co op erative
customs that could allow develop ers to attract co-develop ers and get maximum leverage out of the medium.
But what is this leadership style and what are these customs? They cannot b e based on p ower
relationships { and even if they could b e, leadership by co ercion would not pro duce the results we see.
Weinb erg quotes the autobiography of the 19th-century Russian anarchist Krop otkin's \Memoirs of a
Revolutionist" to go o d e ect on this sub ject:
\Having b een brought up in a serf-owner's family,Ientered active life, like all young men of my time, with
a great deal of con dence in the necessity of commanding, ordering, scolding, punishing and the like. But
when, at an early stage, I had to manage serious enterprises and to deal with free men, and when each
10. The So cial Context of Free Software 16
mistakewould lead at once to heavy consequences, I b egan to appreciate the di erence b etween acting on
the principle of command and discipline and acting on the principle of common understanding. The former
works admirably in a military parade, but it is worth nothing where real life is concerned, and the aim can
be achieved only through the severe e ort of many converging wills."
The \severe e ort of many converging wills" is precisely what a pro ject like Linux requires { and the
\principle of command" is e ectively imp ossible to apply among volunteers in the anarchist's paradise we
call the Internet. To op erate and comp ete e ectively, hackers who want to lead collab orative pro jects have
to learn how to recruit and energize e ective communities of interest in the mo de vaguely suggested by
Krop otkin's \principle of understanding". They must learn to use Linus's Law.
Earlier I referred to the \Delphi e ect" as a p ossible explanation for Linus's Law. But more p owerful
analogies to adaptive systems in biology and economics also irresistably suggest themselves. The Linux
world b ehaves in many resp ects like a free market or an ecology, a collection of sel sh agents attempting to
maximize utility which in the pro cess pro duces a self-correcting sp ontaneous order more elab orate and
ecient than any amount of central planning could achieve. Here, then, is the place to seek the \principle
of understanding".
The \utility function" Linux hackers are maximizing is not classically economic, but is the intangible of
their own ego satisfaction and reputation among other hackers. One may call their motivation \altruistic",
but this ignores the fact that altruism is itself a form of ego satisfaction for the altruist . Voluntary
cultures that work this way are not actually uncommon; one other in whichIhave long participated is
science ction fandom, which unlike hackerdom explicitly recognizes \egob o o" the enhancement of one's
reputation among other fans as the basic drive b ehind volunteer activity.
Linus, by successfully p ositioning himself as the gatekeep er of a pro ject in which the development is mostly
done by others, and nurturing interest in the pro ject until it b ecame self-sustaining, has shown an acute
grasp of Krop otkin's \principle of shared understanding". This quasi-economic view of the Linux world
enables us to see how that understanding is applied.
Wemay view Linus's metho d as an way to create an ecient market in \egob o o" { to connect the
sel shness of individual hackers as rmly as p ossible to dicult ends that can only b e achieved by
sustained co op eration. With the fetchmail pro ject I have shown alb eit on a smaller scale that his
metho ds can b e duplicated with go o d results. Perhaps I haveeven done it a bit more consciously and
systematically than he.
Many p eople esp ecially those who p olitically distrust free markets would exp ect a culture of self-directed
egoists to b e fragmented, territorial, wasteful, secretive, and hostile. But this exp ectation is clearly falsi ed
by to give just one example the stunning variety, quality and depth of Linux do cumentation. It is a
hallowed given that programmers hate do cumenting; how is it, then, that Linux hackers generate so much
of it? Evidently Linux's free market in egob o o works b etter to pro duce virtuous, other-directed b ehavior
than the massively-funded do cumentation shops of commercial software pro ducers.
Both the fetchmail and Linux kernel pro jects show that by prop erly rewarding the egos of many other
hackers, a strong develop er/co ordinator can use the Internet to capture the b ene ts of having lots of
co-develop ers without having a pro ject collapse into a chaotic gang-bang. So to Bro oks's LawI
counter-prop ose the following:
19: Provided the development coordinator has a medium at least as good as the Internet, and knows how to
11. Acknowledgements 17
lead without coercion, many heads are inevitably better than one.
I think the future of free software will increasingly b elong to p eople who knowhowtoplay Linus's game,
p eople who leave b ehind the cathedral and embrace the bazaar. This is not to say that individual vision
and brilliance will no longer matter; rather, I think that the cutting edge of free software will b elong to
p eople who start from individual vision and brilliance, then amplify it through the e ective construction of
voluntary communities of interest.
And p erhaps not only the future of free software. No commercial develop er can match the p o ol of talent
the Linux community can bring to b ear on a problem. Very few could a ord even to hire the more than
twohundred p eople who have contributed to fetchmail!
Perhaps in the end the free-software culture will triumph not b ecause co op eration is morally rightor
software \hoarding" is morally wrong assuming you b elieve the latter, which neither Linus nor I do , but
simply b ecause the commercial world cannot win an evolutionary arms race with free-software communities
that can put orders of magnitude more skilled time into a problem.
11 Acknowledgements
This pap er was improved by conversations with a large numb er of p eople who help ed debug it. Particular
thanks to Je Dutky
formulation and help ed develop ed the analysis that pro ceeds from it. Also to Nancy Leb ovitz
Perceptive criticisms also came from Joan Eslinger
con ict b etween GPL and the bazaar mo del. I'm grateful to the memb ers of PLUG, the Philadelphia
Linux User's group, for providing the rst test audience for the rst public version of this pap er. Finally,
Linus Torvalds's comments were helpful and his early endorsementvery encouraging.
12 For Further Reading
I quoted several bits from FrederickP. Bro oks's classic The Mythical Man-Month b ecause, in many
resp ects, his insights haveyet to b e improved up on. I heartily recommend the 25th Anniversary addition
from Addison-Wesley ISBN 0-201-83595-9 , which adds his 1986 \No Silver Bul let" pap er.
The new edition is wrapp ed up byaninvaluable 20-years-later retrosp ective in which Bro oks forthrightly
admits to the few judgements in the original text whichhave not sto o d the test of time. I rst read the
retrosp ective after this pap er was substantially complete, and was surprised to discover that Bro oks
attributes bazaar-like practices to Microsoft!
Gerald P.Weinb erg's The Psychology Of Computer Programming New York, Van Nostrand Reinhold
1971 intro duced the rather unfortunately-lab eled concept of \egoless programming". While he was
nowhere near the rst p erson to realize the futility of the \principle of command", he was probably the
rst to recognize and argue the p oint in particular connection with software development.
13. Version Id : cathedr al paper:sg ml ; v 1:241997=09=1522 : 48 : 58esr E xpesr 18
Richard P. Gabriel, contemplating the Unix culture of the pre-Linux era, argued for the sup eriorityofa
primitive bazaar-like mo del in his 1989 pap er Lisp: Good News, Bad News, and How To Win Big . Though
dated in some resp ects, this essay is still rightly celebrated among Lisp fans including me . A
corresp ondent reminded me that the section titled \Worse Is Better" reads almost as an anticipation of
Linux. The pap er is accessible on the World Wide Web at .
De Marco and Lister's Peopleware: Productive Projects and Teams New York; Dorset House, 1987; ISBN
0-932633-05-6 is an underappreciated gem whichI was delighted to see Fred Bro oks cite in his
retrosp ective. While little of what the authors havetosay is directly applicable to the Linux or
free-software communities, the authors' insightinto the conditions necessary for creativework is acute and
worthwhile for anyone attempting to imp ort some of the bazaar mo del's virtues into a more commercial
context.
13 Version
Id : cathedr al paper:sg ml ; v 1:241997=09=1522: 48 : 58esr E xpesr