Scientific A Status Report on its Adoption in the HEP Environment

Alan Silverman (on behalf of HEPiX Board) 15th June 2005

For many years, most HEP labs have used the free Redhat release of Linux as the basic for their compute farms and their desktop systems. Major exceptions were DESY (SuSE) and GSI1 ().

In the summer of 2003 Redhat announced that they would split their release into the free, open source, community-supported Fedora source distribution and the licensed Enterprise Linux (RHEL) release. Further the distribution of the latter in binary for-free form would be discontinued, and the company would focus on the commercial "Enterprise" distribution. However the open source nature of Linux means they must still publish the GPL2 source used to build their distribution and that must remain free. While adoption of the free Fedora release may look attractive to sites wishing to remain with a free distribution, Redhat stated that Fedora would undergo rapid (around 6 monthly) version change and Fedora releases would not be supported for much longer than the previous release (9-12 months)3. On the other hand, RHEL releases would benefit from up to 5 years support, the kind of support timescale desired by enterprises, including HEP production sites.

By the autumn of 2003 the US DoE had negotiated a deal for their sites and SLAC and Jefferson Lab signed up but FNAL and BNL decided against this. CERN started to try to negotiate a similar deal to the US DoE, not only for itself but also for its associated labs. Redhat appeared sympathetic but despite contacts with the HQ staff looking after our market, these talks did not result in an interesting offer.

Having decided against using the RHEL licence, FNAL started from the freely- available (because open source, see above) Redhat RHEL source and built what it called at that time LTS. FNAL made this available to all HEP sites and even included modules to permit local tailoring. In parallel with the Redhat negotiations, CERN adopted a similar procedure to Fermilab, starting from the RHEL source and building its own release, which it called CEL.

Just before Spring HEPiX 2004 in Edinburgh (May 2004) Redhat proposed an agreement to CERN to make a deal for all HEP sites similar to the one offered to the US DoE and indeed they presented this to the HEPiX meeting in open session. But by then, FNAL and CERN were committed to their respective RHEL source-based solutions and these also were also presented at HEPiX, alongside each other. Given the similarity of the two solutions, the fact that the two teams were already interacting with each other and under the prompting of many sites presented, Fermi and CERN agreed to collaborate on a uniform "HEP Linux" distribution (which nevertheless could be tailored to some extent for each site). A strong selling argument would be the binary compatibility with Redhat’s RHEL distribution. Several other labs represented declared themselves very interested in adopting this. HEP Linux was renamed to Scientific Linux (SL) shortly afterwards, CERN's versions are SLC3, Fermi's versions are SLF3. CERN switched its local certification to SLC3.

1 For the purposes of this report, we consider the HEPiX community although strictly speaking GSI is not generally considered an HEP lab. 2 Gnu Public Licence 3 In fact in the past few days Redhat have cut Fedora loose and it is only supported now by the Linux community. Over the following months, CERN completed SLC3 certification and started migrating all central systems to this. In parallel, it nevertheless signed up to a deal with Redhat to evaluate their licensed RHEL offering (actually called subscriptions) on a limited number of nodes (200) and also to evaluate Redhat support (the so-called TAM support which is also used by SLAC).

At the autumn HEPiX in BNL many labs reported taking up SL (or the SLF or SLC builds) and all reported being very happy with it. Further, they all reported good binary compatibility between SL and RHEL, for example BaBar uses RHEL at SLAC but SLf in their Italian sites. DESY announced that they will phase out SuSE in favour of SL over the coming year. Regarding the licensed offering and Redhat support, SLAC reported being more or less satisfied with their Redhat support (value for money compared to the cost of personnel which would be needed if they did not have TAM support) and CERN said it was too early to judge.

At the recent Spring 2005 HEPiX in Karlsruhe there were yet more reports of successful take-up of SL (including SLF and SLC) and it is now clearly the HEP standard Linux with only two known exceptions (SLAC and GSI), although there are a few sites where other flavours of Linux can be found for specific purposes or evaluations.

Looking to the future, FNAL have announced a release of SL4 based on Redhat’s RHEL version 4 although their production systems are still based on SL3 and they expect to publish appropriate fixes to SL3 for some time yet. Although FNAL have stated formally that they do not support SL for other sites, they have established a support infrastructure (web site, mail list, etc) which facilitates community support within the HEP world and which they naturally lead (although they would welcome and encourage other sites to participate more).

CERN is still considering its policy with respect to moving to SLC4 or SLC5 because of the timing when a fully certified release of either could be deployed and the subsequent support lifecycle with respect to the startup timing of the LHC (scheduled for the middle of 2007). Regarding Redhat support, CERN currently report mixed feelings - some issues take too long to get resolved inside the company – but on balance will probably continue it for the moment at least.

The bottom line is that Scientific Linux rules almost all of the HEP world. The major exception is SLAC which looks like staying with the licensed RHEL solution but compatibility between the two does not look to be an issue.

Why was Scientific Linux a success? At Spring 2005 HEPiX, Jan Iven of CERN put forward a number of possible reasons • Last year many sites were in a position ready to move forward and all were affected by changes in Redhat’s licence policy • Sites preferred to remain if possible with the Redhat flavour • Slow progress in negotiating an acceptable contract with Redhat (apart from some labs who decided to take up an offer to DoE sites – only SLAC?) • The potential for SL (and consequently SLC) to be compatible with RHEL was attractive along with built-in flexibility for local tailoring • No forced commercial contracts needed • The example of major sites adopting it with the underlying implication that they would fix problems (although neither FNAL or CERN was in a position to guarantee support)

In my opinion, for this happy state of affairs, we should be grateful to • FNAL Linux Support team for making a distribution which is freely and easily usable and adaptable to other sites and for supporting this, even if unofficially • CERN Linux support for basing their distribution on the same foundation • HEPiX for offering a platform for publicising the ease of use and the success of SL/SLF/SLC.

Now the challenge for the mid-term future is to keep the HEP community focused on a single version of SL, with differing deadlines and constraints pulling the labs forward or holding them back. The emerging 64bit architectures and their associated compilers are also factors that could promote divergence over time.