Comparison of code review systems for OpenJDK Draft version: May 27th, 2011 Cross referencing request for features and use cases with capabilities in Review Board and Crucible

Review Board Crucible Webrev Links to home pages http://www.reviewboard.org/ http://www.atlassian.com/ software/crucible/tour/ Feature / Requirement Comments Actions Post code review requests Yes - GUI and CLI (extensions) Yes - GUI and CLI (extensions) Yes, by email Crucible: http://confluence.atlassian.com/display/FECRUDEV/Integration+Tutorials ; Review Board: http://www.reviewboard.org/docs/manual/dev/webapi/ Republish a request for review Yes - GUI and CLI (extensions) Yes - GUI and CLI (extensions) Yes, by email Should preserve old history View all diffs since repo baseline No Yes, using Fisheye View diff from the repo since last review No Yes, using Fisheye View all diffs on the repo since review request No Yes, using Fisheye Post delta reviews, or review of a review Upload diff in TEXTAREA and point to a hg changeset url Yes Yes Ability to dynamically add metadata to reviews (on the fly) Can write a plug-in to support this Adding attributes to the reviews like testing, security, performance meta data to the review Create a review for code that has already been checked in Yes (using post-review) Yes (post commit reviews) Create a review for code that is yet to be checkin Yes Yes Comment on a review Yes Yes Download a patch for the proposed change Yes Yes To use additional tools -- for example, build and test the change for myself. Track which comments have been addressed and which have not by the owner of No Can classify comments on reviews with the code/content ‘defect’ tags and create workflows to manage them Ability to have discussions and brainstorming over a change, before a review request for a fix is submitted. And then link to the communication when the formal review is presented in the system Group a bunch of review requests to one review request Iterate over review/fix/review/fix workflow till the fix is committed Yes Yes

Users Declare interest to accept reviews in an area of interest Yes Yes Disapprove triggers a notification to team lead and or manager No Can build it as a workflow based on a review classification Select from a list of approvers Yes Yes, very flexible mechanism for managing reviews based on roles, projects, arbitrarily etc. Allow approvers to signup to an review Yes Setup with project permissions Nominate reviewers Yes Yes Accept reviews and comments from everyone Yes, can setup and control with permissions Overriding approver. No Yes. Can create a moderator role Tune out reviews and comments from non-reviewers Yes Yes

Code views Flexibility in viewing code changes and diffs No (side-by-side only) Yes (side-by-side or unified) The reason to need different diff file formats is that, different types of diffs are easier to visualize in different diff formats. In visual code review systems, the multiple views are achieved by use of colors and font characteristics. Provides various diffs views (cdiff, udiff, frames etc...) as provided by webrev http://www.atlassian.com/software/fisheye/ Different diff tour/ and http://www.atlassian.com/ formats software/fisheye/tour/enterprise-dvcs.jsp

Notifications Subscribe to notifications in an area of interest No Yes, through dashboards http:// www.atlassian.com/software/crucible/ tour/code-review-reports.jsp Opt in and out of email notifications Yes Silent modification to the review request (no email sent) Email notifications of review requests Yes Yes Opt in for notifications, even if one is not a reviewer Yes Notification subscription on a repo No Yes (opt in for commit notifications) RSS feeds to notifications and activity No Yes (doc) Email notify author that fix is approved Yes Email notification of updates, nag mail, etc. Yes Yes Tracking system: what reviews do I have outstanding, who has approved my Yes Yes, very powerful dashboard. And can outstanding reviews so far, etc. integrate with bug system.

Interface View diff side by side, raw diffs are not sufficient Yes (side-by-side) view of different Yes. http://www.atlassian.com/software/ versions fisheye/tour/ and http://www.atlassian.com/ software/fisheye/tour/enterprise-dvcs.jsp Ability to interact with the CR system from an IDE Eclipse plugin Yes (IntelliJ and also Eclipse plugin available) Command line interface to the CR system Limited (post-review) Limited (but possible to write script using REST API) wget patch file without authentication Ability to issue review requests from the command line Yes (post-review) No (but possible to write script using REST API) command line retrival of diffs and review contents to try out the diffs in build and test No Can write a script/extension for this using the API Initiate a review by email and perform CR transactions over email No No

History History of changes around a line of code No Yes, using Fisheye History and relationship between a line of code, change requests(CRs), actual changes(change log), Archive all review requests Yes Yes

Offline Ability to do review code diffs offline Nice to have feature

Patches Able to generate 'patch' of diffs that can later be applied using gpatch (or some other tool) Provide various diffs views (cdiff, udiff, frames etc...) as provided by webrev http://www.atlassian.com/software/fisheye/ tour/enterprise-dvcs.jsp

Repositories Publish a review that proposes a change to multiple repositories (open, closed, No No. (it is a proposed enhancement for sustaining releases, 1.4.2, 5, 6 etc) Crucible/Fisheye) Add a extra repo to the review No Yes Deal with nested repositories well (forest or subrepos) No No Work well with Mercurial Yes Yes Review target Source code Yes Yes Formatted and raw HTML Documents in text, rtf, etc Text docs ok (as long hg diff can be Text docs ok (as long hg diff can be used) used)

Extensibility API to extend the system Yes (REST API) Yes (REST API and Java API)

Integration Integration with bug system Yes (by associating bug tracker to repo) Yes (JIRA) Integration with the code versioning system Yes (repositories setup) Yes ( through FishEye) Link an approved review to the actual change set. A review is just a proposal, the No Yes ( through FishEye) actual committed change needs to be linked to the CR when it is done. Ability to extract details (escalation id, responsible engineer etc..) from the Bug Can link to bug id’s in another system. Tight integration to JIRA (http:// Systems (OpenJDK bug system) when we put in the bug-id into the tool But no tight integration www.atlassian.com/software/crucible/ tour/jira-issues-code-review.jsp)

Archiving Permanent store of reviews Yes Yes Permanent storage of reviews and discussions on them Yes Yes

Releases Publish a review that proposes the same change to multiple releases No connection to releases. No connection to releases, but can be connected to issues(sub-issues in JIRA Add a extra release to the review No connection to releases. No connection to releases, but can be connected to issues(sub-issues in JIRA Migration Ability to perform batch move of all reviews out of our current tool code review tools to the new tool. webrev "webrev" is fine. JDK team has been using webrevs for many years. Don't change it! - - +