Distributed Development with Bazaar

Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Distributed development with Bazaar

Yann Pouillon

1. Facultad de Químicas, Universidad del País Vasco UPV/EHU, Donostia-San Sebastián, Spain. 2. European Theoretical Spectroscopy Facility (ETSF), Spain.

CECAM Tutorial — Zaragoza, Spain 2010/06/21 Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Outline

1 Overview An evolutionary process Quick tour of Bazaar 2.1

2 An extremely ﬂexible tool Keeping up with an evolving project Popular Bazaar workﬂows

3 Example: the management of ABINIT Sustaining the growth of the code The ABINIT Forge Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Outline

1 Overview An evolutionary process Quick tour of Bazaar 2.1

2 An extremely ﬂexible tool Keeping up with an evolving project Popular Bazaar workﬂows

3 Example: the management of ABINIT Sustaining the growth of the code The ABINIT Forge Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Different times, different challenges, different models

1970’s: first steps (SCCS times) VCS = tool to keep history of single files weave method: uniform data retrieval time outcome: SCCS adopted by Unix 1980’s: from solo to team work (RCS times) independent streams of development project management & VCS decoupled outcome: data retrieval efficiency & reliability 1990’s: organizing collaboration (CVS times) golden age of client-server model uprising of the centralized model outcome: atomicity, file management, deployment 2000’s: going back to freedom emergence of the distributed model explosion of "ecological diversity" outcome: better flexibility, usability & scaling Overview An extremely flexible tool Example: the management of ABINIT The end

Different times, different challenges, different models

The centralized model

Strength: control Individualistic (driving force = project leader) Requires a server Requires permanent connection to the server Snapshotting & publishing tightly bound

Associated social model: benevolent dictatorship

Subversion: "subvert the beast" Since 2000: overcome CVS limits while keeping compatibility Today: probably the most popular for the centralized model Reason: smooth transition from CVS Overview An extremely ﬂexible tool Example: the management of ABINIT The end

The centralized model

Strength: control Individualistic (driving force = project leader) Requires a server Requires permanent connection to the server Snapshotting & publishing tightly bound

Associated social model: benevolent dictatorship

The centralized model

Strength: control Individualistic (driving force = project leader) Requires a server Requires permanent connection to the server Snapshotting & publishing tightly bound

Associated social model: benevolent dictatorship

The distributed model

Strength: freedom & autonomy Team-oriented (driving force = community) Exploration of new concepts made easy (no loss of history) Network access: only to publish or access remote changes

Associated social model: doocracy the actual developers take the decisions dynamical reorganization, delegation of powers mostly bottom-up structure

Bazaar: "VCS for Human Beings" "Just a tool" philosophy + bindings to other VCS’s Overview An extremely ﬂexible tool Example: the management of ABINIT The end

The distributed model

Strength: freedom & autonomy Team-oriented (driving force = community) Exploration of new concepts made easy (no loss of history) Network access: only to publish or access remote changes

Associated social model: doocracy the actual developers take the decisions dynamical reorganization, delegation of powers mostly bottom-up structure

Bazaar: "VCS for Human Beings" "Just a tool" philosophy + bindings to other VCS’s Overview An extremely ﬂexible tool Example: the management of ABINIT The end

The distributed model

Strength: freedom & autonomy Team-oriented (driving force = community) Exploration of new concepts made easy (no loss of history) Network access: only to publish or access remote changes

Associated social model: doocracy the actual developers take the decisions dynamical reorganization, delegation of powers mostly bottom-up structure

Bazaar: "VCS for Human Beings" "Just a tool" philosophy + bindings to other VCS’s Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Centralized vs. distributed: a false debate

Former bone of contention

Patch bomb = unmanageable contribution Distributed model: no protection Centralized model: just more difﬁcult

Patch bomb = social failure =⇒ no technical solution Importance of ﬂuid communication Interplay between social & technical aspects Centralized model popular for historical reasons Balance between control & freedom: YOU choose! Reminder: CVS = beast of SVN −→ SVN = beast of other tools Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Centralized vs. distributed: a false debate

Former bone of contention

Patch bomb = unmanageable contribution Distributed model: no protection Centralized model: just more difﬁcult

In conclusion ...

Do not feed the troll! Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Outline

1 Overview An evolutionary process Quick tour of Bazaar 2.1

2 An extremely ﬂexible tool Keeping up with an evolving project Popular Bazaar workﬂows

3 Example: the management of ABINIT Sustaining the growth of the code The ABINIT Forge Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Main objectives of Bazaar

Usability: "DVCS for Human Beings" (Ubuntu philosophy) Portability: only requires Python ≥ 2.4 Efficiency: smart support for renaming files & directories optimal use of disk space minimal overhead (fast branching & merging) Productivity: "just a tool" (read the documentation) Extensibility: Python plugins & well-documented API Pluggability: suitable for embedded systems Safety: strong & active open-source community test suite with > 10,000 automatic tests Freedom: Bazaar is Free Software (GPLv2+) Overview An extremely flexible tool Example: the management of ABINIT The end

Bazaar core concepts

Revision: snapshot of the state of a ﬁle tree Branch: ordered series of revisions Repository: store of revisions (branches sharing history) Working Tree: version-controlled directory holding source Checkout: source tree remotely connected to a branch Standalone Tree: independent branch with working tree Branching: action of splitting development lines Merging: action of joining development lines Conﬂict: two branches contain different changes at the same place Criss-Cross: painfully intricate branches

For more information http://doc.bazaar-vcs.org/bzr.dev/en/user-guide/ http://doc.bazaar-vcs.org/bzr.dev/en/user-reference/bzr_man.html Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Bazaar core concepts

The Bazaar working tree

Directory structure my_project/ +-.bzr/ +-README +-branch/ +-branch-format +-branch-lock/ +-checkout/ +-repository/ +-.bzrignore +-file1 +-file2 +-file3 +-... Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Bazaar main commands

bzr help: get generic help bzr help commands: get help on commands bzr [command] --help: get help on [command] bzr init: create a standalone tree bzr status: display current status of a branch bzr add: tell Bazaar to version-control files & dirs bzr remove: tell Bazaar to stop version-controlling files bzr commit: take a snapshot of the working tree bzr push: publish history to a remote branch bzr pull: import history from a remote branch bzr info: display branch information Overview An extremely flexible tool Example: the management of ABINIT The end

Committing changes

Writing changelogs One short summary line (no trailing dot)

* dir1/sub1/file1: Some changes. Make full sentences.

* dir2/sub2/file2,dir3/sub3/file3: Some other changes. * dir4/sub4/file4: Related changes (no blank line before).

* Additional notes and issues.

Recommended: GNU changelog format All lines < 80 characters and start at ﬁrst column Reference document: http://www.gnu.org/prep/standards/html_node/Change-Logs.html Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Merging branches

When joining development lines: conﬂicts may happen Solving conﬂicts: graphical tools might help, e.g.: kdiff3 xxdiff

bzr merge: merge a branch into another −→ criss-cross =⇒ bzr merge --weave bzr conflicts: display remaining conflicts bzr resolved: mark a conflict as resolved bzr revert: rollback files/everything to the last snapshot Overview An extremely flexible tool Example: the management of ABINIT The end

Merging branches

When joining development lines: conﬂicts may happen Solving conﬂicts: graphical tools might help, e.g.: kdiff3 xxdiff

Good practice

Committing safely ALWAYS issue a bzr status before committing

Merging safely

1 Commit first 2 Merge the other branch 3 Solve the conflicts 4 Commit again 5 Continue developing ... Overview An extremely flexible tool Example: the management of ABINIT The end

Good practice

Committing safely ALWAYS issue a bzr status before committing

Merging safely

1 Commit first 2 Merge the other branch 3 Solve the conflicts 4 Commit again 5 Continue developing ... Overview An extremely flexible tool Example: the management of ABINIT The end

User interfaces

GUI: Bazaar Explorer Web: Loggerhead Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Outline

1 Overview An evolutionary process Quick tour of Bazaar 2.1

2 An extremely ﬂexible tool Keeping up with an evolving project Popular Bazaar workﬂows

3 Example: the management of ABINIT Sustaining the growth of the code The ABINIT Forge Overview An extremely ﬂexible tool Example: the management of ABINIT The end

There is no "one true way"

Deﬁnition Workﬂow: way a team chooses to work together

In a project: workﬂow changes may become necessary With Bazaar: the tool adapts to your needs Purpose: let you focus on what truly matters

Following workﬂow descriptions Extracted from the Bazaar User Guide http://doc.bazaar-vcs.org/bzr.dev/en/user-guide/ Overview An extremely ﬂexible tool Example: the management of ABINIT The end

There is no "one true way"

Deﬁnition Workﬂow: way a team chooses to work together

In a project: workﬂow changes may become necessary With Bazaar: the tool adapts to your needs Purpose: let you focus on what truly matters

Following workﬂow descriptions Extracted from the Bazaar User Guide http://doc.bazaar-vcs.org/bzr.dev/en/user-guide/ Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Outline

1 Overview An evolutionary process Quick tour of Bazaar 2.1

2 An extremely ﬂexible tool Keeping up with an evolving project Popular Bazaar workﬂows

3 Example: the management of ABINIT Sustaining the growth of the code The ABINIT Forge Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Solo / single

Best practice for programming / writing: backup of old versions rollback to an earlier state tracking of history fast and lightweight

Papers, posters, talks, reviews, ... Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Partners / peer-to-peer

Advantages over solo: easier sharing of data line-wise tracking of changes (who did what) no server required intelligent merging Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Lock-step / centralized

Advantages over centralized VCS’s: better branching and merging simpler renaming support (files & directories) ability to change workflow whenever it makes sense Overview An extremely flexible tool Example: the management of ABINIT The end

Centralized with local commits

Advantages over centralized: can work ofﬂine (e.g. during travel) less interference of bad commits Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Distributed with shared mainline

Advantages over centralized with local commits: easier organization of work cross-merging possible shared repositories and integrated merging Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Distributed with human gatekeeper

Advantages over distributed with shared mainline: systematic review of the code tighter control over when changes go to mainline Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Distributed with automatic gatekeeper

Advantages over distributed with human gatekeeper: systematic testing of the code (higher mainline integrity) better scaling with team growth dedicated tool: Patch Queue Manager (PQM) Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Outline

1 Overview An evolutionary process Quick tour of Bazaar 2.1

2 An extremely ﬂexible tool Keeping up with an evolving project Popular Bazaar workﬂows

3 Example: the management of ABINIT Sustaining the growth of the code The ABINIT Forge Overview An extremely ﬂexible tool Example: the management of ABINIT The end

The mutation of ABINIT

Challenges Growing size of the code Growing complexity of the code Growing number of contributors Growing size of the community

ABINIT developers: all around the world + heterogeneous Until 2004 source code managed by CVS workflow: mostly solo 2004 −→ 2008 transition: CVS −→ GNU Arch −→ Bazaar exploration of workflows Overview An extremely flexible tool Example: the management of ABINIT The end

Breaking free from the client-server model

GNU Arch (2001) Originally: set of speciﬁcations to manage project history −→ generic protocol −→ overcome the drawbacks of centralized versioning −→ freedom regarding implementations

Speciﬁcally designed for distributed development Work while disconnected from the network Branching and mirroring do not require cooperation Uses available infrastructure: no dedicated server support for multiple network protocols permissions managed by the underlying system Implementations in C / C++ / Python Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Breaking free from the client-server model

GNU Arch (2001) Originally: set of speciﬁcations to manage project history −→ generic protocol −→ overcome the drawbacks of centralized versioning −→ freedom regarding implementations

From GNU Arch to Bazaar

TLA (Tom Lord’s Arch) [2001–2005] home page: http://wiki.gnuarch.org/FrontPage several forks: ArX, arch++, baz part of the GNU Project in 2003 basic maintenance since 2006

Baz (fork of TLA) [2004–2005] project initiated by Canonical Ltd. (Ubuntu) simpler UI, new archive format, better algorithms many issues: usability, lack of ﬂexibility, bad scaling

Bazaar "New Generation" (bzr) [since 2005] home page: http://bazaar-vcs.org/ not based on GNU Arch anymore, still using some concepts completely rewritten from scratch in Python 2.4 key component of Ubuntu and Launchpad Overview An extremely ﬂexible tool Example: the management of ABINIT The end

From GNU Arch to Bazaar

TLA (Tom Lord’s Arch) [2001–2005] home page: http://wiki.gnuarch.org/FrontPage several forks: ArX, arch++, baz part of the GNU Project in 2003 basic maintenance since 2006

Baz (fork of TLA) [2004–2005] project initiated by Canonical Ltd. (Ubuntu) simpler UI, new archive format, better algorithms many issues: usability, lack of ﬂexibility, bad scaling

From GNU Arch to Bazaar

TLA (Tom Lord’s Arch) [2001–2005] home page: http://wiki.gnuarch.org/FrontPage several forks: ArX, arch++, baz part of the GNU Project in 2003 basic maintenance since 2006

Baz (fork of TLA) [2004–2005] project initiated by Canonical Ltd. (Ubuntu) simpler UI, new archive format, better algorithms many issues: usability, lack of ﬂexibility, bad scaling

Outline

1 Overview An evolutionary process Quick tour of Bazaar 2.1

2 An extremely ﬂexible tool Keeping up with an evolving project Popular Bazaar workﬂows

3 Example: the management of ABINIT Sustaining the growth of the code The ABINIT Forge Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Components of the Forge

File server available through SSH backup facility merging made easy Repositories one per developer / team containing all necessary branches strict naming conventions for public branches Maintenance utilities creation of repositories & branches information about archive status management of permissions (Unix) Test farm / nightly builds checking branch status: buildability & mergeability monitoring critical ﬁles building branches on various architectures −→ various hardware / OS / compiler combinations reporting nicely & daily online Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Components of the Forge

Different scales, different workﬂows

Top level: distributed with human gatekeeper Geographical teams: peer-to-peer, lock-step, local commits Topical teams: peer-to-peer, distributed + shared mainline Maintainers: solo, using shared repositories

Example of shared repository ABINIT/ +-6.3/ # <--- Repository +-6.3.0-pouillon/ +-6.3.0-trunk/ +-6.3.0-gmatteo-public/ Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Beneﬁts to ABINIT from Bazaar

Central archive for data storage & sharing Reduced overhead for merges Easy propagation of partial merges All developer teams may work as they prefer Only initial support required (autonomy comes fast) Services: test farm, branch browser, web reporting

Good practice A technical tool never fulfills social duties To ensure sustainability: define & enforce clear rules for developers define & enforce clear policies for the branches make sure information flows Overview An extremely flexible tool Example: the management of ABINIT The end

Acknowledgments

Funding by MICINN & ETORTEK ABINIT Developers Bazaar Developers Wikimedia Commons & Wikipedia art.gnome.org Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Always remember ...

Golden rule of versioning Commit early, commit often!

Thank you! Overview An extremely ﬂexible tool Example: the management of ABINIT The end

Always remember ...

Golden rule of versioning Commit early, commit often!

Thank you!