Mirada (transl., Spanish feminine noun): A look or gaze, especially one of particular meaning or interest.
Mid-resolution InfRAreD Astronomical Spectrograph
System Design Note: Notes on Mercurial (hg) Usage
S. N. Raines
On behalf of the MIRADAS Consortium
2013-01-08 058_001 mi_058_001_hg_notes Full Contents
Full Contents ii
Executive Summary v
Useful Mercurial References vii
1 Orientation 1 1.1 Mercurial Interfaces ...... 1 1.2 File Concepts ...... 1 1.2.1 Recommendations for LATEX document text ...... 2
2 Required Mercurial Reading 3 2.1 Other Useful References ...... 4 2.1.1 Obtaining help ...... 4 2.1.2 Three reference cards ...... 4 2.1.3 Manuals ...... 4
3 Mercurial Installation 9 3.1 Mac Installation ...... 9 3.2 Windows Installation ...... 9 3.3 Linux Installation ...... 9 3.3.1 Check installed Python ...... 9 3.3.2 Acquire code ...... 9 3.3.3 Installation ...... 9 3.3.4 Add hg to path ...... 10 3.3.5 Test the install ...... 10 3.4 Set up .hgrc file ...... 10
4 Example Mercurial Structure and Use 13 4.1 Instlab Structure ...... 13 4.2 My Workflow ...... 13 4.2.1 Update the master repository ...... 14 4.2.2 Keep a USB drive copy ...... 14 4.3 Examples with commands ...... 14 4.3.1 Working with others ...... 15 4.4 What Files are Tracked? (.hgignore)...... 15
ii Processed on: January 8, 2013 at 14:42.
058_001 iii / 17
Executive Summary
This brief document will point to some useful on-line guides for using Mercurial, a distributed version control system tool. Mercurial may be used for any text document, be it source code or non-software, e.g. manuals.
v
Useful Mercurial References
Steve Borho. TortoiseHg, 01 2013. URL http://tortoisehg.bitbucket.org/. Adrian Buehlmann. A Tutorial on Using Mercurial, 2012. URL http://mercurial. selenic.com/wiki/Tutorial. Adrian Buehlmann. Understanding Mercurial, 2012. URL http://mercurial. selenic.com/wiki/UnderstandingMercurial. Adrian Buehlmann. Learning Mercurial in Workflows, 2012. URL http://mercurial. selenic.com/guide. Adrian Buehlmann. Other Tools, 01 2013. URL http://mercurial.selenic.com/ wiki/OtherTools. Steve Losh. hg tip, 2013. URL hgtip.com/tips/beginner. Matt Mackall. hg Mercurial source code management system, 2012. URL http://www. selenic.com/mercurial/hg.1.html. Mercurial, Arne Babenhauserheide, Steve Losh, and David Soria Parra. Mercurial, 2012. URL http://mercurial.selenic.com/. William Stafford Noble. A Quick Guide to Organizing Computational Biology Projects. PLoS Computational Biology, 5(7):e1000424, July 2009. doi: 10.1371/journal.pcbi.1000424. g001. URL http://dx.plos.org/10.1371/journal.pcbi.1000424.g001. Bryan O’Sullivan. Mercurial: The Definitive Guide, 2009. URL http://www.selenic. com/mercurial/hg.1.html. Bryan O’Sullivan. hgrc Configuration files for Mercurial, 2012. URL http://www. selenic.com/mercurial/hgrc.5.html. Joel Spolsky. Hg Init: a Mercurial tutorial, 2012a. URL http://hginit.com/. Joel Spolsky. Hg Init: Ground Up Mercurial, 2012b. URL http://hginit.com/01. html.
vii
1 Orientation
Mercurial is a Python-based1 distributed version control system (DVCS)2. In a DVCS anyone can obtain a complete working copy of a project (including the entire version control history), make changes to the project and publish them as a separate fork, or submit them back to the main project for merging. I am not an expert on version control software, but from what I have read (cf. Spolsky, 2012a) many people find that merging contributions from many sources is much easier in a DVCS such as Mercurial than in a centralized version control system such as CVS, RCS, or SVN. This results from how Mercurial and Git track changes—it keeps a list of changes to every file at every ‘commit’; these ‘change-sets’ are kept in a hidden (.hg) folder.
1.1 Mercurial Interfaces
Mercurial is natively a command-line application which is used from a terminal on either Mac OS X or on a Linux host. It is also widely used in the ’cloud’ for collaborative sharing of code, for example at bitbucket.org. Several GUI interfaces exist for Mac (e.g. MacHg, SourceTree); TortoiseHg is very good and a highly recommended one for Window platforms that is reported to work on Linux (Buehlmann, 2013), although I have not attempted to install it on my desktop Linux computer.
1.2 File Concepts
As noted above, Mercurial and Git operate by keeping track of every version of every file. You can tag a commit of a change-set at any point, to aid your ability to go back in history to examine its contents3. But, you only ever need one filename for each file. You never, ever, should need to manually create extra snapshot files within the repository—avoid the temptation to keep, e.g. file.txt, file-2013-01-07.txt, file-2013-01-07b.txt, etc. All this will do is make your list of tracked files in the directory harder to keep track of mentally (cf. Spolsky, 2012b; Noble, 2009). DCVSs were originally created for managing software, where the file contents are commonly very structured, generally with short lines (∼72 characters). When obtaining
1Python versions 2.4 through 2.7 are supported 2Another, more widely used DVCS is Git. 3Also, since Mercurial contains the entire history, you can have multiple, parallel, “sandboxes” for prototype development. But this requires a good understanding of Mercurial and keeping good track mentally of what you’re doing with all these different copies.
1 1.Orientation
new changes from others or from a master project, it is possible to see the file differences with your files, on a line-by-line basis. But, DCVSs may work on any text-based file, such as a LATEX document. Hence it is key to maintain an agreed-upon standard for how lines are entered. If only editing a small portion of text, it is important to not change the formatting of entire blocks, as doing so will make a large number of unnecessary ‘diffs’ where no content has actually changed.
1.2.1 Recommendations for LATEX document text This suggests two options for .tex files: 1. Hard-wrap every line of the document to an agreed-upon line width, say 72 charac- ters, or 2. hard-wrap every sentence. It will be easier to see commit differences using (2), as a single sentence is the funda- mental ‘atom’ of a thought. This is discussed, for example, by Andrew Stacey at heading StageTwo:LineWrapping of http://www.math.ntnu.no/~stacey/HowDidIDoThat/ LaTeX/versioning.html. Similarly, the standard style rules could have special LATEX environments on single lines. It is key that your LATEX editor does not automatically hard wrap lines as you type. In Emacs this is called Auto Fill; it is called Word Wrap in Text Modes in Emacs 22 or Auto Fill in Text Modes in Emacs 24 in the menu button at top. TeXShop ‘soft’ wraps long lines at spaces between words, as the line approaches the edge of the editing window. This make editing text very readable. With Emacs 22 However earlier you can have it truncate lines longer than the window, or you can have it ‘soft’ wrap the text. However these Emacs 22 and earlier wrap text in the middle of word, with little curled arrows at the end of the visual line and start of the next line, which affects text readability when editing a document. In contrast, Emacs 24 has a new mode called ‘Word Wrap (Visual Line Mode)’. Emacs 24 is not installed system-wide on the Linux hosts, however I have installed a version on my scratch drive, for my own use.
2 / 17 058_001 2 Required Mercurial Reading
I learned much about using Mercurial by installing it and playing with it on personal, stand-alone, projects, after reading several very clear introductions available online. I suggest you read these two introductions, in this order, if you have never worked with a DCVS: • Hg Init: a Mercurial tutorial, at hginit.com (Spolsky, 2012a). This cov- ers the basics, including sharing change sets, and is a good start if you have never worked with a DCVS, and progresses in this order: – Subversion Re-education (skip if you’re new to version control) – Ground up Mercurial – Setup for a Team – Fixing Goofs – Merging – Repository Architecture • Learning Mercurial in Workflows, at mercurial.selenic.com/guide/ (Buehlmann, 2012). The basic workflows include – Log keeping – Lone developer with nonlinear history – Separate features – Sharing changes while the advanced workflows include – Backing out bad revisions – Collaborative feature development – Tagging revisions – Removing history The advanced workflows can be read as you get familiar with Mercurial. After reading those, then the main Mercurial website has two links that are very useful: • Understanding Mercurial, at http://mercurial.selenic.com/wiki/UnderstandingMercurial (Buehlmann, 2012), and
3 2.Required Mercurial Reading
• A Tutorial on Using Mercurial, at http://mercurial.selenic.com/wiki/Tutorial (Buehlmann, 2012). This one has ten (10) parts that include coverage of – Installation Mercurial – Initializing a repository – Cloning an existing repository – Navigating the history of a repository – Making changes to a repository, and – Sharing, Exporting, and Merging changes – Resolving failed merges.
2.1 Other Useful References
You can obtain information about Mercurial from the command line, and there are many other useful references online.
2.1.1 Obtaining help If you do not remember, or want more information, about a command you can obtain help: hg: Typing hg on the command line will result in listing of 17 basic commands. hg help: Will result in a complete list of commands. hg help
2.1.2 Three reference cards I have found three reference cards online, reproduced in the following pages. I have used the first one, from Mercurial, and the second one from Adrian Sai-wah Tam, to learn about various workflows. The third reference card, from BBD, nicely lists the basic Mercurial commands by their type of action.
2.1.3 Manuals • Mackall (2012) is the main manual page, reproduced online. All of the command line options are described here. • Losh (2013) has short blog entries on how to use different aspects of Mercurial. • O’Sullivan (2009) is the on-line copy of a book-length manual.
4 / 17 058_001
maintenance Recover recover from an interrupted commit or pull Verify verify the integrity of the current repository
Mercurial is a modern distributed version control system and can be used via the GUI or command prompt. These easy to remember commands provide the full set of functionality available in the command prompt. ALM www.bbd.co.za
3 Mercurial Installation
A few pointers for installing Mercurial on a Mac, Windows, or Linux host.
3.1 Mac Installation
Mercurial 2.4.2 for MacOS X 10.6–10.8 is available from Mercurial et al. (2012). I did not have any problems installing it.
3.2 Windows Installation
I installed TortoiseHg (Borho, 2013) on my ancient Windows XP computer without diffi- culties. I have only played with it for a very short amount of time. The interface is very functional, but I think that understanding the concepts of working on a command line before using it can help make it easier to use (without committing a change you did not want).
3.3 Linux Installation
Department home space accounts are quota limited to about 1 GB. Mercurial is small, on the order of 10 MB, and should be installed directly in one’s home space. To install, follow the notes on http://mercurial.selenic.com/wiki/UnixInstall, along with the comments below; the character > represents the command line prompt.
3.3.1 Check installed Python > python --version
The result should be in the range of 2.4–2.7; on barrayar this returns Python 2.5.1.
3.3.2 Acquire code > wget http://mercurial.selenic.com/release/mercurial-2.4.1.tar.gz
3.3.3 Installation > tar xvf mercurial-2.1.1.tar.gz
9 3.Mercurial Installation
Then use make to install. Note the default make install-home failed for me, and I instead did
> make install PREFIX=/astro/homes/
Also, in my home directory I also had pre-existing bin and lib directories; I do not recall if they will be created if they do not exist.
3.3.4 Add hg to path The Mercurial command line executable is hg. It should be installed in $[HOME]/bin/, which should be in your PATH. If not, edit your .cshrc file to include it. Similarly the python libraries will be installed to $[HOME]/lib/python. Modify your .cshrc file:
setenv PYTHONPATH $HOME/lib/python setenv PYTHONPATH $[PYTHONPATH]:$[HOME]/lib/python2.5/site-packages/
3.3.5 Test the install $ hg debuginstall # sanity-test the install Checking encoding (UTF-8)... Checking extensions... Checking templates... Checking patch... Checking merge helper... Checking commit editor... Checking username... No problems detected
If you get complaints about missing modules, you probably haven’t set PYTHONPATH correctly.
3.4 Set up .hgrc file
For command line use on Mac and Linux, you will want to tell Mercurial who you are and several of your user preferences. These are kept in a file in ~/.hgrc. Mine looks like this:
[ui] username = Nick Raines
[diff] git = True
[extensions] color =
10 / 17 058_001 Set up .hgrc file graphlog = prompt = ~/Computing/hg-prompt/prompt.py progress =
You want all of these except for prompt, unless you obtain prompt.py from me. O’Sullivan (2012) describes the .hgrc file in detail.
058_001 11 / 17
4 Example Mercurial Structure and Use
We need to be able to keep a safely backed-up main repository of MIRADAS systems engi- neering LATEX files, which compile cleanly to PDF output without errors. Simultaneously, we need to have similarly backed-up ‘sandbox’ repositories for concurrent modification of said files; once the changes have been approved they should be pulled and merged within the main repository.
4.1 Instlab Structure
On instlab1, I have created a test structure in Documents/, presently populated with directories for • common files (MI_0012_Common_master/), • functional performance requirements (MI_0012_FPRs_master/), • the OCDD (MI_0013_OCDD_master/), and • the FPRD (MI_0014_FPRD/master/). Each of these folders is its own Mercurial repository 2. Additionally, within Documents/ is a staging directory MI_staging/; everyone working with these files should place their backed-up working copies here. For example, it has a directory snr/ for my working files, which contains: • snr_MI_0012_Common/ • snr_MI_0012_FPRs/ • snr_MI_0014_FPRD/
4.2 My Workflow
All of my working files are edited on my laptop, chalion; when I’m closing up for the day (or have made a major change), I log in to my desktop Linux computer, barrayar, and pull the new changes to my sandbox. My workflow model is as follows:
1/astro/data/instlab/INST_GROUP/PROJECTS/MIRADAS/ 2 The files within MI_0012_Common_master/ are required by all of the other LATEX documents, e.g. by the OCDD, FPRD, etc.. The common master directory will change occasionally, say as new aliases are added, otherwise few changes will occur here.
13 4.Example Mercurial Structure and Use
1. Initialize the master Mercurial repositories in Documents/; populate and commit with files. 2. Clone master files to my laptop (chalion). 3. Make edits on my laptop. Frequently compile to output PDF, fix compilation errors. 4. Log in to my desktop (barrayar), and cd to my staging directory (MI_staging/snr/). 5. Pull changes from my laptop to desktop. 6. Verify desktop version also compiles to generate output PDF. The final PDF is not tracked by Mercurial—more on this later. 7. Occasionally pull changes from others with folders within staging; either pull to my staging area or to my laptop (synchronize both at some point). 8. Occasionally see if there are new changes from the master (i.e. use hg incoming).
4.2.1 Update the master repository Since I control write access to the master directory, I can cd there and pull changes from any of the staging directories. After pulling in new change sets and verifying the content compiles, I should send email to everyone notifying them that they should pull the updates to their staging area sandboxes.
4.2.2 Keep a USB drive copy When I’m working at home it is not always easy or convenient to pull my latest commits from my laptop to my office desktop computer. Hence, I almost always have third, concurrent clone of my laptop’s Mercurial repository on a USB drive. A key to this functional model is to always import to the USB drive, and keep it with you—don’t lose it.
4.3 Examples with commands
A listing of the commands I would use for the above workflow are given here. Clone master to laptop (from laptop): hg clone ssh://nraines@barrayar//astro/data/instlab/ INST_GROUP/PROJECTS/MIRADAS/Documents/MI_0012_Common_master snr_MI_0012_Common Note this format clones the master to a new folder with a different name. Look for changes on laptop: hg status Add any new files on laptop: hg add Commit new changes on laptop: hg commit Note the default editor used for commits is vi/vim. It is possible to change the editor choice in your .hgrc preferences. When committing, try to keep the commit message descriptive but only one line long. Look at the log: hg log or hg glog This will show you all of the commits in order, showing who committed at a particular date and time. The short one-line commit message will be shown. If you
14 / 17 058_001 What Files are Tracked? (.hgignore)
want more details, say about the last commit, use hg log -v -r 1; this wil print the full commit message and a list of the files committed with this change-set. Log in to barrayar and look for incoming changes to my sandbox: From within Documents/snr/snr_MI_0012_Common/ execute hg incoming –stat ssh://raines@chalion//Users/raines/Documents/snr_MI_0012_Common/ This will show you how many new change-sets exist, and the –stat flag will show a diffstat-style summary of changes to each of the changed files. You can also see the actual file differences if you add the -p or –patch option. If you are concerned about pulling the incoming changes, then maybe you should make a parallel, temporary, clone of your sandbox. Pull changes to barrayar: If the changes looked good, execute hg pull ssh://raines@chalion//Users/raines/Documents/snr_MI_0012_Common/
Update working directory on barrayar: After pulling a change-set the changes are in the .hg repository, but the working directory with the editable files still shows your files prior to the pull. You need to execute hg update to obtain a working copy. Then attempt to recompile the LATEX, and examine the file differences. If you imported from someone else’s repository and something’s wrong with their content, nows the time to talk.
4.3.1 Working with others This model of pulling changes from your laptop to your staging area should work seam- lessly, as you’re essentially working as a lone developer. But when working with others it is possible to have merge conflicts if several people have edited the exact same section of text. At that point Mercurial will tell you to merge with the hg merge command. Mercurial can attempt to make the merge itself, but you’ll want to tell Mercurial to use a merging editor, i.e. one that shows you the two conflicting versions of a file. It is possible to setup your .hgrc file to use Emacs for this—I still need to learn how to do so; on a Mac you can use TexWrangler for difference editing—I also need to setup my .hgrc on chalion to use it properly for this task. The key way to avoid merge conflicts with our LATEX documents is to ensure we are working on different sections of a document prior to pulling changes.
4.4 What Files are Tracked? (.hgignore)
Mercurial will track every file within the working directory, but it is probable that you have files that could be ignored. Mercurial works best by tracking changes to text files, but it can and will track differences to binary files, such as png or pdf files which we commonly use for figures. However, every time a binary file changes the .hg repository will potentially grow in size by the size of the binary file. My thought is that the images used in the figures will not change very often as text is written; it is more important to track the files used for figures, as doing so eliminates any need for thought when one pull a change-set—any new binary files that you do not have will be transfered with a pull, without one needing to manually copy them.
058_001 15 / 17 4.Example Mercurial Structure and Use
Note, the final PDF could also be tracked, but this is a bad idea. The final PDF should be regenereated before every commit, which can lead to a large number of changes to an ever increasing in size binary file. This is needless, as the LATEX document should be compilable within the sandboxes and in the master repository. Thus, I recommend that we (a) do track binary files used for figures, but (a) do not track the final pdf, as it can and will be regenerated many times. Hence every repository can have a .hgignore file, which itself is tracked. My default ignore file for LATEX projects looks like this, and ignores these auxilliary files: barrayar[MSDN-051-099-SNR/MSDN_058_hg_notes]> more .hgignore syntax: glob
.DS_Store *.aux *.blg *.dvi *.lof *.log *.lot *.out *.synctex.gz(busy) *.synctex.gz *.tex~ *.toc
syntax: regexp ^(.*\.DS_Store)$ ^mi_058_001_hg_notes.pdf$
The final line in the example shows the pdf for this very document is ignored. It also ignores standard LATEX auxilliary files, list files for figures, tables, etc., and any Emacs backup files; the DS_Store files are a creation of Mac OS X and are best ignored as shown here.
16 / 17 058_001 Document Construction Information
This document was created with pdflatex, available in current up-to-date latex distribu- tions (specifically the TexLive). The memoir document class was used; this class provides a number of carefully-designed document structures. Examples in the class manual showed how to produce a document which should fit on both US Letter and A4 papers. Should have 17 pages.
17