uoi orè,Pu alc,Kna isn jt rn,RcroWurmus Ricardo Prins, Pjotr Hinsen, Konrad Garlick, Paul Courtès, Ludovic 7Fbur 2020 February 17 ciiyRpr 2018–2019 Report Activity a Temp: 22 deg C deg 22 Temp: Supplier: ACME Supplier: b b = 30ml = b a = 10ml = a erdcbesfwr elyetfrhg-efrac computing. high-performance for deployment software Reproducible 20. 2. 19. https://hpc.guix.info/about https://hpc.guix.info/blog/2019/02/guix-hpc-activity-report-2018/ 65 9 https://gitlab.com/makhlaghi/reproducible-paper http://tourbillion-technology.com/ 64 8 https://uthsc.edu/ 7 https://www.cray.com 6 https://www.cnrs.fr/en 5 https://ubc.uu.nl/ 4 https://www.mdc-berlin.de/ 3 https://www.inria.fr/en/ 2 https://guix.gnu.org 1 . inception its after commits 40,000 than more and 10 yamjrmlsoe h ees nMy21 fGUGi .,svnyears seven 1.0, Guix GNU of 2019 May in release the milestone: major a by eraon oa,eray22.hserwasmarked 2020.Thisyear today,February agoand year a previousreport 9 hsrpr ihihskyaheeet fGi-P ewe our between Guix-HPC of achievements key highlights report This . Technology Tourbillion and 8 (UTHSC), Center Science Health Tennessee of University , the Inc. Cray, 7 6 , CNRS including organizations, and individuals additional from tributions 5 N uxfrHCadrpouil cec a eevdcon- received has science reproducible and HPC for Guix . GNU (UBC) ter 4 etro MolecularMedicine(MDC) Centerfor adtheUtrechtBioinformaticsCen- ,and 3 etpoetivligtrersac nttts Inria institutes: research three involving project ment h a Delbrück Max the , 2 uxHCaluce nSpebr07s on softwaredevelop- joint September2017asa in Guix-HPCwaslaunched ihrpouil research. reproducible with ist aei etrto o P rciinradscientistsconcerned practitionersand HPC for tool better a it make to aims HC.Gi-P ulsuo h N Guix GNU the upon builds Guix-HPC (HPC). otaedpomn oland tool deployment software 1 elyett cetfcwrfosadhg-efrac computing high-performance and workflows scientific to deployment uxHCi olbrtv fott rn erdcbesoftware reproducible bring to effort collaborative a is Guix-HPC ! ideas your hear to love we’d and do can we lot a There’s 65 otk datg fGuix. of advantage take to template paper ducible 64 aeenwaeloneetdi dpigMhma Akhlaghi’srepro- Mohammad adapting in samevein,wearealsointerested 18. 3. 4. 17. Outline Perspectives Guix-HPC aims to tackle the following high-level objectives: Making Guix more broadly usable on HPC clusters remains one of • Reproducible scientific workflows. Improve the GNU Guix tool set to bet- our top priorities. Features added this year to guix pack are one way to ter support reproducible scientific workflows and to simplify sharing approach it, and we will keep looking for ways to improve it. In addition to and publication of software environments. this technical approach, we will keep working with cluster administrators • Cluster usage. Streamlining Guix deployment on HPC clusters, and to allow them to deploy Guix directly on their cluster. We have seen more providing interoperability with clusters not running Guix. cluster administratorsdeploy Guix this year and we are confident that this trend will continue. • Outreach & user support. Reaching out to the HPC and scientific re- search communities and organizing training sessions. Last year, we advocated for tight integration of reproducible deploy- ment capabilities through Guix in scientific applications. The GNU Guix Workflow Language and Guix-Jupyter have since matured, giving us more The following sections detail work that has been carried out in each insight into the benefits of the approach and opening new perspectives of these areas. that we will explore. We would additionally like to investigate a comple- mentary approach: adding Guix support to existing tools, such as jupyter- repo2docker62. For the Guix Workflow Language we will continue to explore its suit- ability in scheduler-less compute environments, such as ad-hoc clusters of short-lived virtual servers, that are becoming increasingly popular. We think thatthepropertiesof bit-reproduciblebuildsand package-levelgran- ularity unlock hitherto unavailable sharing among independent parts of workflow environments to an extent that is impossible when using mono- lithic container images. This increase in storage and deployment efficien- cy is expected to result in significant cost savings when computations are offloaded to externally hosted and metered resources. We have witnessed increasing awareness in the scientific communi- ty of the limitations of container-based tooling when it comes to building transparent and reproducible workflows. We are happy to be associated with the “Ten Years Reproducibility Challenge”63 where we plan to demon- strate how Guix can help reproduce computational experiments. In the 62 https://repo2docker.readthedocs.io/en/latest/ 10 63 https://hpc.guix.info/blog/2019/05/gnu-guix-1.0-foundation-for-hpc-reproducible-science/ https://rescience.github.io/ten-years/ https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pull.html 13 https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-describe.html 12 https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-environment.html 11 woemi s aei paigGi) u snwspotdi more a in supported now is but Guix), updating is case use main (whose tp4ue orqiearte nnutv omof form unintuitive rather a require to used 4 Step . describe uxpull guix 12 13 uoaie.Se sspotdby supported is 2 Step automatized. tp3by 3 , step uxenvironment guix guix 11 tp1i eystaindpnetadcnteeoentb fully be not therefore can and situation-dependent very is 1 Step .Rpouigacmuainfo uharecord. a such from computation a 4. Reproducing structed. .Poiigarcr fteeednisrmwihte a berecon- can they which thedependenciesfrom of record a 3. Providing rgasfo h niomn htae“utthere”. “just are that environment the from programs .Esrn htteeaen idndpnece,sc sutility as such dependencies, hidden no are there that 2. Ensuring .Fnigtedpnece facomputation. a of dependencies the 1. Finding teh iifraisCne UC:1pro-er(olJanssen) (Roel person-year 1 (UBC): Center Bioinformatics • Utrecht pects: er(famFahe n jt Prins) Pjotr and Flashner (Efraim year uprigrpouil opttosrqie drsigfu as- four addressing requires computations reproducible Supporting nvriyo ense elhSineCne UHC:08person- 0.8 (UTHSC): Center Science Health Tennessee of • University lmy n ouetto,bt rcia n akrud a scarce. was background, and practical both documentation, and clumsy, opttos h sritraet erdcblt etrswsabit a was features reproducibility to interface user the computations: nvriéd ai:02 esnya SmnTournier) (Simon person-year 0.25 Paris: de • Université hr eetoosalst sn uxfratal on reproducible doing actually for Guix using to obstacles two were there orilo ehooy . esnya Pu Garlick) (Paul person-year 0.7 Technology: • Tourbillion uxhsawy upre erdcbecmuain ydsg,but design, by computations reproducible supported always has Guix lnInlPatra¸scu) Ionel alin ad RcroWru n M and Wurmus (Ricardo ˘ ˘ a ebükCne o oeua eiie(D) person-years 2 (MDC): Medicine Molecular for Center Delbrück • Max etrSpotfrRpouil Research Reproducible for Support Better ae esc,Emne glo dinGuilbaud) Adrien Agullo, Emmanuel Felsoci, Marek otiuost h uxHCcanl lrn rvs,Gle Marait, Gilles Pruvost, Florent channel: Guix-HPC the to contributors aemd ute rgest nueGi drse hsuecase. use this addresses Guix ensure to progress further made have ni:2pro-er LdvcCuts arc rmn,adthe and Brémond, Maurice Courtès, (Ludovic person-years 2 • Inria: rdc n npc h otaeniomnsftoeeprmns We experiments. those softwareenvironmentsof the inspect and produce NS .5pro-er(ordHinsen) (Konrad person-year 0.25 • CNRS: nii ehd ebleeta rrqiiefrti steaiiyt re- to ability the is this for prerequisite a that believe We method. entific rjc,wihw umrz here. summarize we which project, xeiet—oa’ a oeok—skyt salsigargru sci- rigorous a establishing to key notebooks—is lab experiments—today’s inspect reproduce fGi-P,priiaigisiuin aeddctdwr or othe to hours work dedicated have institutions participating Guix-HPC, of computational and to ability The Guix-HPC. for goal hn6 epevr ot— 0 nraeoprdt atya.Aspart year. last to increasecompared 50% month—a peopleevery 60 than uprigrpouil eerhi eea ean a major remains general in research reproducible Supporting N uxi olbrtv fot eevn otiuin rmmore from contributions receiving effort, collaborative a is Guix GNU Personnel erdcbeSinii Workflows Scientific Reproducible 16. 5. 6. 15. straightforward way by guix time-machine14, which providesdirect access • ARAMIS Plenary Session on Reproducibility, May 201956 (Ludovic to older versions of Guix and all the packages it defines. Courtès) A post on the Guix HPC blog15 explains how to perform the four • JCAD, Oct. 201957 (Ludovic Courtès) steps of reproducible computation, and also explains how Guix ensures • SciClojWeb Meeting, Jan. 202058 (Ludovic Courtès) bit-for-bit reproducibility through comprehensive dependency tracking. • FOSDEM, Feb. 202059 (Ludovic Courtès, Efraim Flashner, Pjotr Prins) Reproducible Deployment for Jupyter Notebooks We also organised the GNU Guix Days60, which attracted 35 Guix Jupyter Notebooks16 have become a tool of choice for scientists contributors and ran for two days before FOSDEM 2020. willing to share and reproduce computational experiments. Yet, nothing in a notebook specifies which software packages it relies on, which puts Training Sessions reproducibility at risk. 61 Together with Pierre-Antoine Rouby as part of a four-month intern- The PRACE/Inria High-Performance Numerical Simulation School ship at Inria in 2018, we started work on Guix-Jupyter17,