Collaborative Software Design & Development Open Source Systems 4

Collaborative Software Design and Development Open Source Systems 4

Collaborative Software Design & Development

Open Source Systems 4

The FreeBSD Project Apache Field Support

Kyle Cullen and Sabrina Smith 2/21/08

The FreeBSD Project: A Replication Case Study of Open Source Development Trung T. Dinh-Trong - Research scientist at Avaya Labs James M. Bieman - Professor at Colorado State University and Director of the Software Assurance Laboratory How open source software works: “free” user-to-user assistance Karim R. Lakhani - Assistant professor in the Technology and Operations Management Unit at Harvard Business School Eric von Hippel - Professor at the MIT Sloan School of Management and Head of the Innovation and Entrepreneurship Group

Advantages Freedom to work on whatever you want The review of others’ works could potentially lead to an increase in your overall software development skill level You can work at your own pace – no deadlines Disadvantages Lack of formal process Poor design and architecture Lack of development tools that are comparable to closed-source industry tools Lack of knowledge of the users’ needs

A study of FreeBSD to determine contributing factors to successful OSS projects Characteristics of previously successful OSS projects Not guaranteed to be necessary or sufficient to future successful OSS projects How? Do the 7 hypotheses presented by Mockus et al. in “Two Case Studies of Open Source Software Development: Apache and Mozilla” hold up for other projects? Specifically, FreeBSD

Email, Bug Reports, and CVS Java program used to search through these data sources and extract relevant information the number of committers the number of deltas committed by each person etc. Questionnaire sent to each of the FreeBSD core developers

Question 1: “What was the process used to develop FreeBSD?” Roles of FreeBSD participants core developers – a small group of senior developers who are responsible for deciding the overall goals and direction of the project committers – developers who have the authority to commit changes to the project CVS repository contributors – people who want to contribute to the project but do not have committer privileges How do individuals identify what they will work on? contributors do what is interesting come up with new feature fix a bug you have found search through list of current bugs and desired features core developers do the rest (mundane tasks, etc.) © 2006, Dewayne E Perry EE 382V – Spring 08 6 Collaborative Software Design and Development Open Source Systems 4 The FreeBSD Project: Questionnaire

Question 1: “What was the process used to develop FreeBSD?” cont. Code Release Deadlines 45 days – integration notifications sent out (15 days left!) 30 days – peer reviews and last minute bug fixes 15 days – code freeze and weekly distributions start (beta versions) 0 days – final release

Question 2: “How many people wrote code for new functionality? How many people reported problems? How many people repaired defects?” 354 committers added code between 1933 and April 2003 337 committers checked in new features 6082+ unique individuals reported problems 224 committers fixed problems

Question 3: “Were these functions carried out by distinct groups of people, that is, did people primarily assume a single role? Did large numbers of people participate somewhat equally in these activities, or did a small number of people do most of the work?” 220/354 did both bug fixing and new feature development 47 committers contributed 80% of the deltas End result: a lot more top committers in FreeBSD than in Apache

Question 4: “Where did the code contributors work in the code? Was strict code ownership enforced on a file or module level?” Contributors were free to work on all parts of code – ownership was not enforced 30% of files were only changed by one developer

Question 5: “What is the defect density of FreeBSD code?” Defect density is similar to that of Apache, and is lower than commercial products.

“Open source developments will have a core of developers who control the code base, and will create approximately 80 percent or more of the new functionality. If this core group uses only informal ad hoc means of coordinating their work, the group will be no larger than 10 to 15 people.” Core group for FreeBSD consisted of 36 people Core group for FreeBSD only wrote 47% of code Edited hypothesis: Core developers (< 15) will control the direction of the project while a larger group of top developers (< 50) will contribute approximately 80% of the code base, and their sum is less than 25% of all developers. Spirit of this edited hypothesis matches the spirit of the original one, so hypothesis is confirmed © 2006, Dewayne E Perry EE 382V – Spring 08 12 Collaborative Software Design and Development Open Source Systems 4 The FreeBSD Project: H2

“If a project is so large that more than 10 to 15 people are required to complete 80 percent of the code in the desired time frame, then other mechanisms, rather than just informal ad hoc arrangements, will be required in order to coordinate the work. These mechanisms may include one or more of the following: explicit development processes, individual or group code ownership, and required inspections.” The high number of top committers in FreeBSD required more mechanisms for coordinating work than were needed in Apache; however, these methods were informal and not enforced. H2 should be edited to suggest that more mechanisms are needed, but they don’t need to be formal.

“In successful open source developments, a group larger by an order of magnitude than the core will repair defects, and a yet larger group (by another order of magnitude) will report problems.” This hypothesis holds for FreeBSD. Could other organizational structures also work? Is a good organizational structure necessary for success?

“Open source developments that have a strong core of developers but never achieve large numbers of contributors beyond that core will be able to create new functionality but will fail because of a lack of resources devoted to finding and repairing defects.” FreeBSD had many contributors, so H4 could not be evaluated.

“Defect density in open source releases will generally be lower than commercial code that has only been feature-tested, that is, received a comparable level of testing.” FreeBSD supports this hypothesis. This hypothesis refers to unstable OSS releases (those that have only been feature-tested), so the authors extend the hypothesis to include stable releases as follows: “Defect density in OSS releases will be lower than commercial code that has only been feature-tested. If an OSS has a mechanism to separate unstable code from stable code or “official” releases, then the defect density of the stable code releases will be equivalent to that of commercial code after release.”

“In successful open source developments, the developers will also be users of the software.” Developers of FreeBSD were also users, so H6 holds true. Can OSS developers who are not users be successful if they listen carefully to user feedback, i.e. is it necessary for OSS developers to also be users?

“OSS developments exhibit very rapid responses to customer problems.” The authors didn’t have access to data about the responsiveness of FreeBSD developments to customer problems, so H7 could not be confirmed.

Are the 7 hypotheses presented here necessary conditions for success? Are they sufficient? Do the hypotheses hold for projects of any size, i.e. can a small OSS project that has only a small number of contributors and bug reporters be successful? Is the method for testing OSS projects more reliable than commercial testing practices?

Why would you develop a software product for free when you could be getting paid? Motivations: User’s direct need for the software and software improvements worked upon Enjoyment of the work itself Enhanced reputation that may flow from making high-quality contributions to an open source project But do these motivations translate to the “necessary-but- mundane” tasks? Not really So how do these tasks get accomplished?

Apache field support is the “necessary-but-mundane” task looked at in this study The Apache Development Group does not provide field support for Apache, so how do users get assistance when there are defects in the program or when they have a general lack of understanding? This study attempts to answer that question and determine what motivates people to voluntarily offer help to those who need it.

The Apache Development Group states: “There is no official support for Apache. None of the developers want to be swamped by a flood of trivial questions that can be resolved elsewhere. Bug reports and suggestions should be sent via the bug report page. Other questions should be directed to the... newsgroup where some of the Apache team... or other HTTPd gurus who should be able to help.” The Usenet newsgroup is the place where Information seekers post questions related to Apache Information providers voluntarily answer those questions for all to see What motivates the information providers to devote their time/resources to help out the information seekers?

Collect usage data from the newsgroup over a period of 4 years (1996-1999) Number of unique questions posted Number of information seekers/providers Percentage of questions asked/answered by individual information seekers/providers Length of time between question being asked and question being answered Etc. Questionnaire sent to each individual who posted a question or an answer from Oct. 1, 1999 to Feb. 15, 2000 Used to gain insight into motives of information seekers and providers

Information Seekers and Providers divided into 4 groups Frequent seekers – seekers who posted 4 or more questions to newsgroup in this time period (10% of seekers) Other seekers Frequent providers – providers who posted 10 or more answers to questions within the time period (10% of providers) Other providers 50% of questions were asked by 24% of seekers 50% of responses were answered by only 2% of providers 50% of questions were answered within 24 hours 25% of questions did not receive responses at all

2 versions of the questionnaire: 1 for seekers 1 for providers Questionnaire focuses on: motivations time spent posting how much work the providers had to do in order to answer the question on the newsgroup Each person only asked to fill out questionnaire once, no matter how many times they posted to the newsgroup If the user answered a seeker questionnaire, but he turned out to actually be a provider more often than a seeker, his questionnaire was thrown out, and vice versa.

From the perspective of the seeker: What is the cost of posting a question? Only the cost of forming and posting the question. (Majority of information seekers – 90.4% – were not posting time critical questions.) Approximately 11.5 minutes to post Would have taken about 115 minutes to find the information on their own Benefit to cost ratio is very good! From the perspective of the provider Cost of answering a question broken into 2 parts An able and willing provider must be matched with a seeker An answer must be provided

Cost of matching a provider to a seeker Cost would be high if providers scanned the newsgroup only to find questions they could answer Actually, providers read newsgroup to gain valuable information about problems others are encountering and how they may be solved. This information is often valuable to the management and upgrading of their own websites. Providers stumble upon questions that they already know the answers to while they are reading, and thus matching providers to seekers has no cost. Cost of answering the question 50% of frequent providers spent < 1 minute answering questions 80% of other providers spent < 5 minutes Reason for low cost is that providers already know the answers Cost is low, but not zero, so why do it? © 2006, Dewayne E Perry EE 382V – Spring 08 27 Collaborative Software Design and Development Open Source Systems 4 Apache Field Support: Provider Motivations

What motivates providers to answer questions posted by seekers? According to the questionnaires, the primary motivations are: The provider expects reciprocity The provider is “helping the cause” The provider will gain reputation or enhance career prospects The provider finds answering questions to be intrinsically rewarding It is the provider’s job (this is the least of the motivations) Are these responses valid? Respondents could only be saying what they believe to be “socially correct”

This study should be relevant to other OSS projects that employ voluntary online support for users, but is that guaranteed to be true? What are the potential issues? The success of Apache field support is largely due to the fact that providers spend a lot of time reading the newsgroup in order to learn. What if we have a project where there is less to learn? Will the newsgroup be as effective if question loads are much greater? Would there be enough providers? What if the questions being asked are unique to the seekers and thus the providers do not have “off-the-shelf” answers for them?

The “necessary-but-mundane” task of field support for Apache was accomplished using voluntary online support. Why were providers willing to give information for free? “The public posting of both questions and answers created a site that potential information providers wanted to visit and study in order to gain valuable information for themselves.” The public posts also included names of the providers, which allowed providers to gain reputation for posting. Providers felt that since they were help by reading the newsgroup, they should help others in kind.

Many providers indicated that they found answering questions to be fun or intrinsically rewarding. In that sense, is the task truly mundane? What do you do with tasks that nobody finds fun? Are there other ways to recruit volunteers to perform mundane tasks? What other OSS-related “necessary-but-mundane” tasks might exist that people would or wouldn’t volunteer for?