Scmbug: Policy-based Integration of Software Configuration Management with Bug-tracking
Kristis Makris Kyung Dong Ryu Department of Computer Science IBM T.J. Watson Research Center and Engineering 1101 Kitchawan Rd Arizona State University Yorktown Heights, NY 10598 Tempe, AZ, 85287 [email protected] [email protected]
Abstract and enhancement requests. Those who do not use a bug-tracking system tend to rely on shared lists, Software configuration management(SCM) and email, spreadsheets and/or Post-It notes to moni- bug-tracking are key components of a successful tor the status of defects. This procedure is usually software engineering project. Existing systems error-prone and tends to cause those bugs judged integrating the two have failed to meet the needs least significant by developers to be dropped or of the ASU scalable computing lab, powered by ignored[10]. open-source software. An improved solution to By examining a log of software changes from the integration problem, designed to accomodate an SCM tool, it is uncertain why the changes oc- both free and commercial systems alike, is pre- curred. By examining a defect report, it is un- sented. certain what changed in software in response to Scmbug offers a policy-based mechanism of the defect. Integration of SCM with bug-tracking capturing and handling integration of SCM ties the reason why a feature/defect was devel- events, such as committing software change- oped/fixed with what software changes occurred sets and labeling software releases, with a in the SCM system to accomplish this. Marry- bug-tracking system. Synchronous verification ing the SCM and bug-tracking systems improves checks and the flexibilty to match multiple devel- the traceability of software changesets, quality of opment models separate this approach from re- documentation in defect reports, and quality of lated work. We address design limitations of ex- release documents. isting integration efforts, suggest improvements SCM[11, 9, 18, 6, 5, 7, 3] and defect- in SCM and bug-tracking systems required to tracking[2, 4, 1] systems are widely popular and achieve a scalable solution, and document our in deployment. Commercial systems implement early integration experiences. a mostly inflexible integration of the two, and free systems lack a variety of important verification 1 Introduction guarantees. None of the existing systems met the integration demands of the ASU scalable com- SCM is key[17] in maintaining quality in soft- puting lab. At a minimum, an integration system ware engineering. SCM systems, or even simple should provide: version control systems, guarantee that a record • Integration of common denominator SCM of all changes and enhancements to software is events, such as committing changesets and maintained. When bugs creep in software, a trace labeling releases. When a changeset is of how changes occurred is available, simplify- committed, the accompanying log message ing the process of identifying regression points should be inserted in the bug-tracker. The and correcting defects. list of affected files, and the older/newer Paired with SCM, bug-tracking is another key version numbers of each file, should be re- in engineering quality software. Defect-tracking flected in the bug report. When a release systems guarantee that nothing gets swept under is labeled, the release version should be in- the carpet; they provide a method of creating, serted in the bug-tracker. storing, arranging and processing defect reports • Synchronous verification checks of SCM
USENIX Association FREENIX Track: 2005 USENIX Annual Technical Conference 11 actions against the bug-tracker. If a devel- changes. Additionally, integration with other oper attempts to commit a changeset against bug-tracking systems requires implementing a an invalid product or bug id, the commit ac- communication interface with the target bug- tivity should fail, and the developer should tracker. Finally, the integration is unsuitable for be informed immediately. deployment across the public Internet, for rea- • Policy-based configuration. The integration sons explained in Section 5.1. system should be able to match the develop- In the free software arena, the most fre- ment model followed by an organization by quently attempted integration involves CVS[11], tuning run-time parameters. the dominant open-source network transparent • Secure deployment of the integration over version control system, with Bugzilla, a free de- the public Internet. fect tracking software that tracks millions of bugs • An interface to integrate any SCM system and issues for hundreds of organizations around with any bug-tracking system, overcoming the world. Steve McIntyre’s website[21] docu- limitations of the involved systems where ments an approach using trigger scripts to email possible. CVS’s actions to an account (Bugzilla Email • A mechanism to produce a Version Descrip- Gateway) configured to parse the email and pro- tion Document (VDD) detailing the defects cess it accordingly. This approach is not syn- corrected, and the changesets involved in chronous. If a user accidentally commits against fixing a defect, between two releases of a the wrong bug number, or a bug against which he software. is not the owner, the SCM system will proceed Scmbug is designed to meet these require- with the commit action regardless. The user will ments. It is implemented in Perl, an excellent, not be given the option to correct her actions. Ad- cross-platform, glue programming language, for ditionally, if the email gateway is not active, the UNIX-like systems, such as Linux, AIX, and So- developer will not be immediately aware that in- laris. It currently supports integration of CVS and tegration failed. Subversion with Bugzilla. The somewhat dated CVSZilla[15] project also integrates SCM events produced by CVS with The rest of this paper is structured as follows. Bugzilla 2.10. It does not support integration of Section 2 outlines the limitations of existing sys- events produced by any SCM system in a generic tems that prompted our solution. The system de- way. Moreover, it modifies the Bugzilla schema, sign is presented in Section 3, and a short exam- and as a result does not work with current ver- ple in Section 4. The system’s components are sions of Bugzilla, such as 2.18. Finally, this in- analyzed in Sections 5, 6, and 7. Section 8 brings tegration is unsuitable for deployment across the to light the improved quality of release documen- public Internet. tation enabled by this system. Early experiences John C. Quillan was first to conceive and im- using Scmbug are documented in Section 9, and plement synchronous verification checks and a insight to future work is given in Section 10. Fi- VDD generator, in work that was never pub- nally, Section 11 concludes this paper. licly released. He integrated CVS with Bugzilla, but his design did not support any SCM sys- 2 Related Work tem with any bug-tracking system, or deployment over the public Internet. Scmbug incorporates In the realm of commercial products, Perforce[7] these features, and proposes a more flexible de- and ClearCase/ClearQuest[3] only offer integra- sign that can accommodate both free and com- tion of their own SCM and bug-tracking sys- mercial SCM and bug-tracking systems. tems, in a proprietary way. They do not read- ily support integration with free SCM systems, 3 System Architecture such as CVS[11], Subversion[9], and Arch[18] or free bug-tracking systems such as Bugzilla[2] Scmbug is a client/server system. As shown in MantisBT[4], and AntHill[1]. Work enabling Figure 1, it consists of a set of SCM system hooks integration of Perforce with Bugzilla 2.0-16 is that capture standard SCM events and a generic available[8], but the integration API lacks an ab- glue mechanism of handling these events on stract bug-tracker interface, and requires mod- the machine hosting an SCM repository. These ifications released by the Perforce integration events are translated into integration requests and team every time the Bugzilla database schema transported to a server daemon. The daemon em-
12 FREENIX Track: 2005 USENIX Annual Technical Conference USENIX Association loginfo, verifymsg, Bugzilla API: taginfo hooks Scmbug globals.pl CVS CGI.pm Bugzilla Integration Glue Daemon activity.pl TCP port 3872 native Subversion Mantis API Mantis pre−commit, post−commit hooks
Figure 1: Scmbug system architecture diagram
ploys a generic mechanism of handling these re- Figure 3 presents the log comment stored in quests and contains functionality that can process CVS, produced in a more readable Changelog these requests per bug-tracking system. format using the cvs2cl tool, in response to The integration glue follows a common in- this changeset. This entry clearly details what terface of handling SCM events, and groups changed: a global variable is introduced to hold the logic driving the behavior of the integra- the name of the network device. However, it does tion. SCM-specific implementation idiosynchra- not explain why this change was required. Why cies, such as parameter decoding from SCM must a global variable be used instead of a hard- hooks, are isolated in separate modules. The ab- coded value? To identify the reason why these straction of an integration glue is what permits files were modified one must consult the bug- any SCM system to be integrated using Scmbug. tracking system. This log entry refers the reader The integration functionality is always executed to bug id 417. on the machine hosting the SCM repository. Figure 4 captures the comment entered in Respectively, the integration daemon imple- Bugzilla in response to the changeset. The origi- ments a common interface of accepting and pro- nal SCM log message is included in the bug com- cessing integration requests. Bugtracker-specific ments. The integration also generates a list of af- functionality, such as reading metadata and up- fected files, matched with the older/newer version dating bug comments, is implemented in separate numbers of each file, and appends it to the log modules. The abstraction of an integration dae- message to show what changed in response to the mon is what permits any bug-tracking system to bug fix. To identify why this changeset occured, be integrated using Scmbug. The SCM reposi- one only needs to scroll in past history in Bugzilla tory and the integration daemon can be hosted on to comment 6, with minimal effort, which reveals separate machines. the root of the problem, as shown in Figure 5: PDAs use wireless, rather than ethernet, for net- work connectivity. The traceability of software 4 Integration example changesets back to the root of problems is dra- matically improved. Lets examine an actual integration sequence us- Note that a complete fix to the bug does not end ing CVS and Bugzilla. Σ-Watch is a perfor- with a single changeset. One day later, the devel- mance monitoring tool created by the ASU scal- oper drafts documentation on how the codebase able computing lab. Monitoring the performance can be configured with different network device of a Zaurus SL-6000 PDA using this tool fails, names. He commits this new document in later and this defect is recorded in bug 417. After some changesets, in comments 13-16. By consulting initial investigation, the failure point is identi- this bug, the entire list of what changesets oc- fied and documented in comment 6. A devel- cured to handle the bug is readily available. oper implements a fix, and attempts to commit his changeset using the command ’cvs commit’. This command brings up a predefined log mes- 5 Integration Daemon sage template, in which the developer explains the source code change and enters the matching Inadequacies of existing systems called for a so- bug id (417) the changeset applies to, as shown in lution based on the abstraction of an integration Figure 2. daemon, as discussed next.
USENIX Association FREENIX Track: 2005 USENIX Annual Technical Conference 13 SCMBUG ID: 417 SCMBUG NOTE: Now using the name of the appropriate network device on each system SigmaWatch runs. This name is set through a global variable.
Figure 2: Documenting a fix for bug 417 during a ’cvs commit’.
2004-08-21 Saturday 11:47 mkgnu
* src/host_node/userspace/server/readnet.c (1.9, p_kpm_prior_to_bug424_fixes), src/host_node/userspace/server/readnet.h.in (1.1, p_kpm_prior_to_bug424_fixes), configure.in (1.33): SCMBUG ID: 417 SCMBUG NOTE:Now using the name of the appropriate network device on each system SigmaWatch runs. This name is set through a global variable.
Figure 3: Log comment entered in CVS for Bug 417, presented in a more readable Changelog format using cvs2cl. Through integration, this CVS entry is also documented in Bug 417, comment 10 of product Σ-Watch in Bugzilla, shown in Figure 4.
5.1 Public Internet Deployment 5.1.1 Minimizing exposure Bugzilla, our defect-tracker of choice, uses First, deployment over the public Internet was re- MySQL as a database backend. Inserting log quired to meet our development model. Mem- messages and version numbers in Bugzilla re- bers of our research group maintain individual quires accessing the database. Opening the SCM repositories on their personal laptops, to al- MySQL TCP port over the public Internet is a low offline development. For our defect-tracking serious security risk, since it exposes access to needs, a publicly available bug-tracking system other applications running on the database sys- is shared. Even though integration with bug- tem. By using a separate integration daemon, and tracking is not possible when a user is working keeping this TCP port closed, only an integration offline, it is still possible to produce the differ- interface is exposed. ences between a repository and a working copy (e.g. using ’cvs diff’), and commit change- 5.1.2 Stand-alone backends sets at a later time. Hence, this method of de- Some bug-tracking systems may not use a velopment is encouraged. When problems arise database backend that listens to a TCP port at all, and development of personal research prototypes such as the Berkley DB backend. Another exam- stalls, pointing each other to a bug-report, for ple is debbugs[16], the bug-tracking software of help, is beneficial. By describing a problem in a the popular Debian Linux distribution. It uses it’s public defect-tracker, rather than a defect-tracker own file-based database and is accessible through running on personal laptops, easy accessibility an email gateway. The abstraction of an inte- and quick resolution of defects are facilitated. gration daemon permits access to such file-based, We also collaborate with researchers from dif- stand-alone backends from the Internet. ferent labs and research organizations on jointly funded projects, and do not wish to permit them 5.1.3 Integration Security access in our LAN using a VPN. Similarly, the Our development model does not match the vi- security policy of collaborators sometimes does sion of existing integration systems, such as Per- not grant us access to their internal resources. Us- force, ClearCase, CVSZilla, or McIntyre’s and ing a public defect-tracker overcomes this prob- Quillan’s integration work. These solutions are lem, and reduces the administrative overhead of deployed in a local area network environment managing separate, private defect-trackers per and directly connect to the bug-tracking database. project or organization. Experience has shown The general assumption is that no malicious users that limited labor and system administration ex- exist in the local network. Since the integration pertise resources exist in an academic environ- daemon listens on a public port, it cannot be as- ment. sumed that all integration requests will originate
14 FREENIX Track: 2005 USENIX Annual Technical Conference USENIX Association ------Additional Comment #10 From Kristis Makris 2004-08-21 11:47 ------
Now using the name of the appropriate network device on each system SigmaWatch runs. This name is set through a global variable.
Affected files: ------1.8 --> 1.9 system/src/host_node/userspace/server/readnet.c NONE --> 1.1 system/src/host_node/userspace/server/readnet.h.in
Figure 4: Bug 417, comment 10 of product Σ-Watch in Bugzilla. The reason why this changeset occured was documented 2 days earlier in comment 6, shown in Figure 5.
------Additional Comment #6 From Kristis Makris 2004-08-19 12:16 ------
The module that collects network information consults the hardcoded "eth0" network device, which does not exist in PDAs. A "wlan0" should be used in them instead, since they have wireless access; not ethernet.
Figure 5: Root of the problem documented in bug 417, comment 6 of product Σ-Watch in Bugzilla. The problem is not documented in CVS.
from trusted developers of our lab. libraries distributed with the Bugzilla codebase The integration communication protocol in- are deployed. The Bugzilla-specific module of cludes the SCM username of the developer com- the integration daemon issues calls to this library. mitting a changeset. This information is needed When the Bugzilla instance is upgraded, the new to identify the developer originating the software Bugzilla codebase includes functionality already change. It is also needed to map to the user- updated by the bug-tracker’s developers to match name of the developer in the bug-tracking sys- it’s database schema. One only needs to point this tem. However, it is possible for an attacker to path variable to the location of the new source produce phony integration requests with prop- distribution and restart the integration daemon. erly matched usernames, and either pollute the Therefore, an upgrade of the bug-tracking sys- bug-tracking system with misleading comments tem does not require upgrading the glue in each or add/delete product version numbers. As a rem- SCM repository using this integration. Addition- edy, public key authentication can be used to con- ally, if a user migrates from Bugzilla to a different firm the identity of integration requests in com- defect-tracking system she only has to specify the munications between the glue and the daemon. name of the new system in the daemon configu- ration file, and restart the daemon. 5.2 Public Integration Interface The bug-tracking system’s codebase does not There’s no public interface that system integra- always provide all the functionality needed by tors can use to communicate with a bug-tracking the verification logic. For example, some queries system. Integration efforts currently duplicate specific to Bugzilla’s database schema had to be functionality implemented in the codebase of the implemented. Informing bug-tracker develop- bug-tracking system. CVSZilla and Quillan’s in- ers of such integration-related queries is impor- tegration work followed this approach. When the tant in providing generic verification logic. The database schema is modified in the next release of Bugzilla developers are now planning[19] to use the bug-tracking system, the integration breaks. XML-RPC to define a public, HTTP-based inter- All SCM repositories must have their glue up- face for integration with 3rd party tools, which dated for the new bug-tracking system in order will include all the functionality needed by Scm- to proceed. bug. Instead, functionality from the bug-tracking Implementation of a similar interface is also system’s codebase is directly reused. For exam- missing in other bug-tracking systems, such as ple, a variable in the integration daemon con- MantisBT and AntHill. Still, integration with figuration file points to the path were the Perl other bug-tracking systems is desirable. Until
USENIX Association FREENIX Track: 2005 USENIX Annual Technical Conference 15 other systems implement a similar interface, the nator, SCM systems are expected today to offer daemon abstraction serves as a mediator, integra- hooks on the following events: tion interface, where custom queries can be im- • pre-commit. Used to verify if the commit plemented. should proceed. 5.3 Lack of SCM Integration Sup- • post-commit. Used to integrate a com- mit log message with a bug-tracking system. port in Bugtrackers Invoked only if pre-commit succeeded. Free bug-tracking systems, such as Bugzilla, • pre-label. Used to verify if a creation of still lack support for SCM integration in their a tag or branch should proceed. database schema. For example, integration work • post-label. Used to integrate the tag must be able to match the username used in or branch name with a bug-tracking system. the SCM system with the username used in Invoked only if pre-label succeeded. the bug-tracking system to enforce a valid bug Scmbug installs scripts in an SCM repository owner check, as described in Section 7.4. A which are executed as hooks for each event. Af- bug-tracking system should provide space in it’s ter information about each event is collected, in- database for each user’s SCM username. Since tegration proceeds through the SCM glue logic. this support is missing in some defect-trackers, Integration of SCM systems that do not provide the need for username mapping is satisfied by the these hooks is not possible. integration daemon. Subversion does not provide the pre-label 5.4 SCM System Limitations and post-label hooks. Tags and branches are created using the ’svn copy’ command The abstraction of an integration daemon also in predefined directories named /tags and makes it possible to develop solutions to some /branches[13]. They are later committed with known integration problems of existing SCM sys- a regular ’svn commit’. Nevertheless, it is tems. still possible to detect during a pre-commit For example, CVS lacks atomic transactions. event if a tag or branch action is underway. If As a side-effect, when the same log message the commit activity indicates that a new subdi- is used to commit files in two separate direc- rectory is created under /tags or /branches, tories, two integration actions occur using the then the transaction corresponds to a labeling ac- same log message. Duplicate log messages are tion. then entered in the bug-tracking system. Com- ing back to the integration example of Section 4, Figure 3 shows that revision 1.33 of the file 7 Integration Policies system/configure.in was produced dur- ing the example changeset. However, Figure 4 Central to the behavior of the common glue logic is missing a corresponding entry for this modifi- is the notion of integration policies. It is essential cation in the affected files list. This change was that the glue prevents developers from describ- documented in comment 11 with a duplicate log ing erroneous integration actions. At the same message. time, the capability to match multiple develop- ment models is equally important. A configura- It is possible to solve this problem by caching tion file stored in the SCM system controls the integration requests at the daemon for a config- overall behavior of the integration glue, as de- urable, short time interval. Reception of another scribed in the next sections. integration request with the same log message indicates that the transaction should have been 7.1 Enabled Integration atomic. The affected files list can be merged, and the log message inserted in the bug tracking sys- The integration can be disabled at any time, by tem only once. changing the value of a flag in the glue config- uration file. Hence, the SCM system can easily back-out from using this integration. 6 SCM Hooks 7.2 Presence of Distinct Bug IDs In modern version control systems, certain SCM SCM systems prompt a developer with a log mes- events can trigger scripts that perform additional sage template prior to committing a changeset. In work handling the event. As a common denomi- order to integrate the log message of a changeset
16 FREENIX Track: 2005 USENIX Annual Technical Conference USENIX Association SCMBUG ID: SCMBUG NOTE: This check ensures developers don’t step on each other’s assigned work. 7.5 Open Bug State Figure 6: Scmbug log message template. Es- Even more important than valid bug owner sential in determining the bug against which a checks is the capability to verify that a change- changeset is commited. set is committed against a bug id set in an open state. For example, committing against a bug with a bug id in the bug-tracking system, a bug id id that has been already resolved, marked either must be included in a parseable format in the log FIXED or INVALID (these are some resolution message. Thus, Scmbug expects the predefined states in Bugzilla), would be wrong. Such ver- template shown in Figure 6 to be filled-in during ification checks alarm the developer that some- a commit action. body else worked on the defect at hand already. Given a log message in this format, the glue Another example would be committing against a logic verifies that one or more bug ids were sup- bug marked as NEW or UNCONFIRMED, where plied. Accepting multiple bug ids is required the CCB has not yet assigned the bug id to a de- when a changeset fixes a collection of defects. veloper. Examples of valid, open states would Additionally, the glue verifies that the ids sup- be ASSIGNED and REOPENED. This verification plied are unique. A duplicate bug id is most likely check ensures that a formal bug dispositioning an indication that a developer typed the wrong process is followed. bug id. 7.6 Valid Product Name 7.3 Valid Log Message Size A product name, specified in the glue configura- Project managers and SCM repository adminis- tion file, is transmitted to the integration daemon trators often have to convince lazy programmers during an integration action. If the supplied bug that there is value in typing a detailed log mes- id is associated with the specified product name sage during commits, and in SCM in general. An in the bug-tracking system, the integration action optional policy that requires the log message to proceeds. This verification check often captures meet a configurable, minimum message size is cases where a developer enters the wrong bug used to address this problem. Messages with a id in a log message in an organization actively size less than the minimum cause an SCM com- developing multiple products at the same time, mit action to be rejected and force a programmer such as Mozilla.org, and the GNOME and KDE to recommit with increased log comment ver- projects. bosity. This check assumes that an SCM system is Simply printing a warning when a small log used to host a single product. In some develop- message is entered and accepting the commit is ment models this may not be true. For example, not enough. A small, incomplete, misleading, or a contracting company may choose to maintain practically useless log comment can cause great documentation of multiple contracts in the same grief when a bug has creeped in the software. SCM system, while assigning different product Such inadequate documentation on a changeset names for them in the bug-tracking system. This introducing the bug can send a developer in a la- approach reduces the administrative overhead of borious SCM history cross-examination and bug- setting multiple SCM repositories. The valid hunting trip. product name policy also permits specification of multiple product names, to match such an alter- 7.4 Valid Bug Owner native development model. Formal defect-tracking processes often define 7.7 Convention-based Labeling a Change Control Board (CCB)[14] or project manager that dispositions bug ids to developers As software evolves through experimental, sta- based on the components they touch. In such ble, or fork stages, the codebase may be labeled settings, developers are not expected to commit accordingly using the SCM system. Common la- changesets that touch bugs assigned to other de- beling needs are: velopers. A policy in Scmbug verifies that the • Releases. The codebase is tagged with a user issuing an integration action is the owner of name indicating that it has reached a stable the bug id specified in the bug-tracking system. state, justifying a release point.
USENIX Association FREENIX Track: 2005 USENIX Annual Technical Conference 17 names => [ # Convention for official releases. gory could be specified. For example, only the re- # For example: lease manager of a product should label releases # SCMBUG_RELEASE_0-2-7 and create forks, a group of high-ranked devel- ’ˆ.+?_RELEASE_[0-9]+-[0-9]+-[0-9]+$’, opers should label developer builds, and all de- # Convention for development builds. velopers should be encouraged to create private # For example: labels. # SCMBUG_BUILD_28_added_policies ’ˆ.+?_BUILD_[0-9]+_.+$’, 8 VDD Generator # Convention for branches. # For example: Release management theory recommends soft- # b_glue_side_policies ’ˆb_.+$’, ware releases be paired with a document de- scribing the changes since the previous re- # Convention for private developer lease. Producing this document directly out # tags. Uses the developer’s initials of the SCM system, either using an auto- # (either 2 or 3). For example: # p_kpm_pre_bug353_fixes mated tool, such as cvs2cl for CVS, or the ’ˆp_[a-zA-Z][a-zA-Z]?[a-zA-Z]_.+$’ SCM system itself, such as the ’svn log -r ]
18 FREENIX Track: 2005 USENIX Annual Technical Conference USENIX Association discovered defects. Another design problem is that older glue in- A tool that can produce such a version descrip- stallations are not preserved in the SCM repos- tion document is under active development. itory. If a newer Scmbug release contains grave defects, one may not easily revert back to an older revision of the integration work. For example, 9 Early Experience CVS versions the hooks themselves. Committing a disabling hook, which is a hook that no longer Scmbug has been deployed in our lab since invokes the glue processing script, still requires March 2004 for integration of various research the original hook invoked for the last time. If prototypes. In hindsight, we were late in devel- this hook is defective, committing will fail. The oping a verbose logging mechanism in the inte- defective hook will not be disabled, reaching a gration daemon. Project deadlines and rapid de- dead-end in upgrading the hook. In CVS one velopment of Scmbug itself discouraged us from must then locally modify the repository to disable continuously upgrading to the latest version. We the hooks using separate RCS commands (RCS are unable to provide statistical information illus- is the underlying database store of CVS). The in- trating the frequency of developer integration er- stallation mechanism can be enhanced to install rors that were caught by the verification policies. permanently in an SCM repository every release of the integration glue, and allow switching be- 9.1 Integration Upgrades tween glue releases. This installation mechanism Seamlessly upgrading the integration work is not will need to be run locally on the machine host- straight-forward. Between release 0.0.8 and 0.1.0 ing an SCM repository, to solve CVS’s dead-end of Scmbug, the communication protocol between hooks problem. The integration daemon can also the glue and the daemon was altered. Upgrading be enhanced to dynamically support older com- the glue suddenly became a multi-stage process: munication protocols. (a) disable the glue, (b) install a newer glue re- 9.2 Case Studies lease, (c) restart the integration daemon, (d) en- able the glue. Disabling and enabling the glue Three undergraduate students in a microproces- was required to accept integration requests from sor systems hardware course worked on one of a glueing codebase that matched the communica- our research prototypes for 16 weeks. They were tion protocol understood by the integration dae- introduced to our lab’s development model, and mon. It was also required for step (b) to suc- a separate CVS branch was created for them to ceed without communication with the daemon. commit their changesets. The students worked Even though an automated glue installation and on a total of 35 bugs. Being new to the concept upgrading tool was employed, manual interven- of SCM, they occasionally checked out the main tion was required on step (c) on the machine run- development line (often referred to as HEAD) in- ning the integration daemon. An important dis- stead of their personal branch. Consequently, advantage is that upgrading multiple repositories they committed some changesets in HEAD instead requires all updated ones to remain without Scm- of their personal branch. A policy for tuning bug integration until all repositories finish steps fine-grained, branch-level, permission checks can (a) and (b), before (c) is carried out. However, ad- solve this problem. For example, committing and ditional scripting by a system administrator could labeling activities could be limited only to our- further automate this process. selves in HEAD, and students could be confined After rapid development of our integration to committing only in their branch. Other devel- work, including rearranging of the Perl packag- opers using Scmbug also requested such a policy. ing structure installed in an SCM repository, the Development of another research prototype multi-stage process was proven to be defective was integrated with Scmbug. Two developers ex- in its implementation. Upgrading from release perienced in the concept of SCM worked in the 0.1.1 to 0.3.1 revealed that the glue libraries used same lab, on 219 bugs, using CVS, for 15 weeks. by the generic SCM hook processing script had Empirically, we can report that the most frequent changed package names and path, and the hooks verification check failed was a valid log message were failing to carry out step (b). The solution to size check for 50 characters. This check correctly this problem was to completely remove the hooks reminded the developers that they should be more rather than disable them through our configura- verbose in documenting changeset history. The tion file. valid bug owner check was the second most fre-
USENIX Association FREENIX Track: 2005 USENIX Annual Technical Conference 19 quent to fail. This was a result of miscommu- the list of affected files in a commit trigger. A nication between developers. We speculate that processing script using a single regular expres- this check may fail more often in open-source sion to parse these arguments gets confused if projects deploying Scmbug: the geographically the filenames contain either commas or whites- distributed nature of developers in such a set- paces. Scmbug handles this issue by employ- ting prohibits them from face-to-face communi- ing a stateful parser, marginalizing the possi- cation on a daily basis. This check, along with bility of getting confused. The single regular the valid product name check, also failed due expression method implemented in McIntyre’s to typing errors while entering the bug id in the bugzilla-watcher script uses only 4 lines log message template. Finally, the least frequent of Perl code for parsing filenames. Our parser verification check to fail was the open bug state was implemented in 145 lines of Perl code, and check. This resulted from developer error in us- required significantly more time to develop. ing TortoiseCVS[12], a GUI CVS client, to com- A plethora of open-source SCM systems mit changesets. This client caches the most re- emerged in the past two years as alternatives cent log messages. Developers often selected one to the currently dominant CVS. Subversion and of the previous messages in order to bring up the OpenCM overcome various limitations of CVS default log message template and type in a new and are still in active development. Arch was message. After clearing out the old log message specifically designed for the distributed develop- and typing a new one, developers occasionally ment needs of open source projects, such as the forgot to change the bug id present in the cached development model followed by the Linux ker- log message. They instead attempted to commit nel, and is still enhancing. Monotone is still in an against resolved bugs. alpha state and has been released only a year ago. We theorize that as these tools are still progress- 9.3 Just In Time Integration ing to a stable state, integration with bug-tracking Within five months of making Scmbug publicly has yet to become a priority in their to-do list, and available, we received over 6300 hits(49 per hence has not been addressed. day) on the project’s webpage[20]. The soft- It is astonishing that mature, high-profile, pub- ware distribution has been downloaded over 3200 lic open-source software development websites times(26 per day). Other system integrators, still lack integration between their SCM and bug- including members of the Bugzilla integration tracking services. Some examples are Source- team, discussed our design and recommended forge (hosting over 93,100 projects, 983,900 policy features which we implemented. Users re- users, in service for 5 years) and GNU Savannah ported deployment of Scmbug to successfully in- (over 2,200 projects, 32,200 users, in service for tegrate both CVS and Subversion with Bugzilla, 4 years). Our solution has been in high demand an activity that is largely simplified via an auto- for a long time. matic installation script. It is clear that this in- tegration tool fills a significant void in the open- source development community. 10 On-Going Work Why wasn’t such an integration system built already? We were unable to find papers doc- OpenCM can commit changesets in disconnected umenting the benefits of integrating SCM with mode[23], a feature also explored by Subversion bug-tracking, or proposing a similar solution. developers. Laptop users can cache a reposi- The Scmbug design is very flexible, and at the tory and work offline. When connectivity is re- same time extremely simple. While the policy stored, they can synchronize their working set mechanism proposed is remarkably useful, it is and resolve conflicts with the main repository. not difficult to implement. Scmbug could be improved to support a discon- One reason might be that limited time is often nected mode of integration. An integration dae- available by developers for work on open-source mon proxy, running on the user’s laptop, could projects. Sometimes, good-enough, quick hacks cache bug-tracking metadata required for policy are easier to implement and preferred, rather than verification checks, such as the list of bug ids, a full-blown, well-designed solution. For exam- their state, bug owner, etc. All integration ac- ple, a common limitation of other systems in- tivity generated by disconnected commits could tegrating CVS with bug-tracking resulted from also be cached and synchronized later with the the inadequate mechanism CVS uses to provide bug-tracker.
20 FREENIX Track: 2005 USENIX Annual Technical Conference USENIX Association The star-topology development model as- to match multiple development models and pro- sumed by Arch increases the significance of in- vide synchronous verification checks. The design tegrating branch and repository names. In this is flexible enough to support any SCM system model, changesets may be produced by dis- with any bug-tracking system, can be deployed tributed SCM repositories, maintained by sepa- over the public Internet, improves the quality of rate development teams. For example, a pub- release documentation, and can overcome limita- lic Bugzilla instance tracks[22] defects in the tions of existing systems. Finally, improvements Linux Kernel. Independent developers and com- in SCM and bug-tracking systems that are critical panies maintaining private forks of the ker- for a successful, generic integration solution are nel post patches to bugs reported by anyone. suggested. However, they do not always report the name and branch of their repository fixing the bug. A user inspecting a bug report is uncertain 12 Acknowledgments which public tree includes the fix. Captur- ing a
USENIX Association FREENIX Track: 2005 USENIX Annual Technical Conference 21 [8] Perforce Defect Tracking Integration. http://freshmeat.net/projects/p4dti/, 2004. [9] Subversion. http://subversion.tigris.org, 2004. [10] The Bugzilla Guide. http://www.bugzilla.org/docs/2.18/html/, 2004. [11] The Concurrent Versions System (CVS). http://www.cvshome.org, 2004. [12] TortoiseCVS. http://www.tortoisecvs.org, 2004. [13] Ben Collins-Sussman, Brian W. Fitzpatrick, and C. Michael Pilato. Version Control with Subversion. O’Reilly Media, 2004. [14] Susan Dart. Concepts in configuration man- agement systems. In Proceedings of the 3rd international workshop on Software config- uration management, pages 1–18, 1999. [15] Tony Garnock-Jones. CVSZilla. http://homepages.kcbbs.gen.nz/˜tonyg/, 2000. [16] Ian Jackson. debbugs. http://www.chiark.greenend.org.uk/ian/debbugs/, 1994. [17] Gregor Joeris. Change management needs integrated process and configu- ration management. In M. Jazayeri and H. Schauer, editors, Proceedings of the Sixth European Software Engineering Conference (ESEC/FSE 97), pages 125– 141. Springer–Verlag, 1997. [18] Tom Lord. Arch revision control system. http://freshmeat.net/projects/archrevctl/, 2004. [19] Kristis Makris. Provide an interface for SCM integration. Bugzilla Bugzilla, ID 255400, 2004. [20] Kristis Makris. Scmbug. http://freshmeat.net/projects/scmbug/, 2004. [21] Steve McIntyre. CVS/Bugzilla integration. http://www.einval.com/˜steve/software/cvs- bugzilla/, 2004. [22] OSDL. Linux Kernel Bug Tracker. http://bugme.osdl.org/, 2004. [23] Jonathan Shapiro and John Vanderburgh. CPCMS: A Configuration Management System Based on Cryptographic Names. In 2002 USENIX Annual Technical Confer- ence, FREENIX Track, 2002.
22 FREENIX Track: 2005 USENIX Annual Technical Conference USENIX Association