Activityreport2019–2020
Total Page:16
File Type:pdf, Size:1020Kb
asDmnkBan uoi orè,PorPis io orir iad Wurmus Ricardo Tournier, Simon Prins, Pjotr Courtès, Ludovic Braun, Lars-Dominik eray2021 February 9 ciiyRpr 2019–2020 Report Activity a Temp: 22 deg C deg 22 Temp: Supplier: ACME Supplier: b b = 30ml = b a = 10ml = a erdcbesfwr elyetfrhg-efrac computing. high-performance for deployment software Reproducible 24. 2. 23. https://hpc.guix.info/blog/2020/02/guix-hpc-activity-report-2019/ 10 http://tourbillion-technology.com/ 9 https://www.cray.com 8 https://leibniz-psychology.org/ 7 https://uthsc.edu/ 6 https://www.cnrs.fr/en 5 https://ubc.uu.nl/ 4 https://www.mdc-berlin.de/ 3 https://www.inria.fr/en/ 2 https://guix.gnu.org 1 eraon oa,eray22.hserwsakdby wasmarked 2021.Thisyear today,February agoand year a viousreport 10 hseotihihseaheeeto Guix-HPCbetweenourpre- Thisreporthighlightskeyachievementsof . ogy 9 nwHE,n TourbillionTechnol- HPE),and (now (ZPID),Cray,Inc. Psychology tutefor 8 7 UHC,teLinzInsti- Leibniz the (UTHSC), Center Science TennesseeHealth of University 6 ,the CNRS organizations,including additionalindividualsand butionsfrom 5 e (UBC) ter N uxfrHCadrpouilsinehsrcie contri- received has reproduciblescience and HPC for Guix GNU . 4 etro MolecularMedicine(MDC) Centerfor adtheUtrechtBioinformaticsCen- ,and 3 etpoetivligtrersac nttts Inria institutes: research three involving project ment h a Delbrück Max the , 2 uxHCaluce nSpebr07s on softwaredevelop- joint September2017asa in Guix-HPCwaslaunched ihrpouil research. reproducible with ist aei etrto o P rciinradscientistsconcerned practitionersand HPC for tool better a it make to aims HC.Gi-P ulsuo h N Guix GNU the upon builds Guix-HPC (HPC). otaedpomn oland tool deployment software 1 elyett cetfcwrfosadhg-efrac computing high-performance and workflows scientific to deployment uxHCi olbrtv fott rn erdcbesoftware reproducible bring to effort collaborative a is Guix-HPC 3. 22. 4. 21. two releases (version 1.1.0 in April 202011 and version 1.2.0 in November12) as well as releases of the Guix Workflow Language and of Guix-Jupyter. 11 https://hpc.guix.info/blog/2020/04/hpc-reproducible-research-in-guix-1.1.0/ 12 https://hpc.guix.info/blog/2020/11/hpc-reproducible-research-in-guix-1.2.0/ https://hpc.guix.info/about 58 ! ideas your hear to love we’d and do can we lot a There’s 58 workflows. asn h a fwa cetsscm oepc ntrso reproducible of terms in expect to come scientists what of bar the raising it ilb ao ou ftecmn er ewn ocnrbt to contribute to want We year. coming the of focus major a be will tists okn ntetosadwrfosdrcl ntehnso scien- of hands the in directly workflows and tools the on Working r indeed are possibility. a workflows osinii oklw.I hwdta ul erdcbersac article research reproducible fully that showed It workflows. scientific to rdcblt hleg”dmntae eycnrtl htGi brings Guix what concretely very demonstrated Challenge” producibility ii ol n oklw.Lkws,orpriiaini h TnYasRe- Years “Ten the in participation our Likewise, workflows. and tools tific thegapbetweenreproduciblesoftwaredeploymentandreproduciblescien- uye,hv aue;aogwt h scNtbo evc,te bridge they service, PsychNotebook the with along matured; have Jupyter, ae oadtoa s ae.Scn,teGi oklwLnug,Guix- Language, Workflow Guix the Second, cases. use additional to cater n oesinii ol;tenwGi cec n uxPs channels Guix-Past and Science Guix new the tools; scientific more and eerhtos is,orpcaecletoshv rw oicuemore include to grown have collections package our First, tools. research ontem o fwr a apndi h rao reproducible of area the in happened has work of lot a Downstream, oigyear. coming fteeareas. these of viaiiyti ilpoal eoeo h anefrsi hsae o the for area this in efforts main the of one be probably will availability;this h olwn etosdti okta a encridoti each in out carried been has that work detail sections following The eest.W aeietfe hlegsrltdt orecd “tarball” code source to related challenges identified have We to. refers h olo civn opeeaciecvrg ftesuc oeGuix code source the of coverage archive complete achieving of goal the erhcmuiisadognzn riigsessions. training organizing and communities search ptem ewl otnet okwt otaeHrtg with Heritage Software with work to continue will we Upstream, urah&ue support. user & Outreach ecigott h P n cetfcre- scientific and HPC the to out Reaching • o-otdpomn ngeneral. in deployment non-root rvdn neoeaiiywt lsesntrnigGuix. running not clusters with interoperability providing oal hog riigssin,adsralnn h s fpcsand packs of use the streamlining and sessions, training through notably lse usage. Cluster temiigGi elyeto P lses and clusters, HPC on deployment Guix Streamlining • hs w opeetr rns ecigott ytmadministrators, system to out reaching fronts: complementary two these hs agtn lsesntrnigGi.W xett otnewr on work continue to expect We Guix. running not clusters targeting those n ulcto fsfwr environments. software of publication and mrvmnssol ewloe by welcomed be should improvements latest The ministrators. e upr erdcbesinii oklw n osmlf sharing simplify to and workflows scientific reproducible support ter uxpack guix erdcbesinii workflows. scientific Reproducible y eaeennraeaotoaditrsaogclustersystemad- interestamong Wehaveseenincreasedadoptionand ty. mrv h N uxto e obet- to set tool Guix GNU the Improve • uxaalblto cetfcoptn lseseanatoppriori- clustersremainsa scientificcomputing availabilityon Guix uxHCam otcl h olwn ihlvlobjectives: high-level following the tackle to aims Guix-HPC Perspectives Outline 20. 5. 6. 19. Reproducible Scientific Workflows Personnel GNU Guix is a collaborative effort, receiving contributions from more Supporting reproducible research in general remains a major than 60 peopleevery month—a 50% increasecompared to last year. Aspart goal for Guix-HPC. The ability to reproduce and inspect computational of Guix-HPC, participating institutions have dedicated work hours to the experiments—today’s lab notebooks—is key to establishing a rigorous sci- project, which we summarize here. entificmethod. Webelievethata prerequisitefor thisistheabilitytorepro- • CNRS: 0.25 person-year (Konrad Hinsen) duce and inspect the software environments of those experiments. • Inria: 2person-years(LudovicCourtès,MauriceBrémond,andthecon- We have made further progress to ensure Guix addresses this use tributorsto theGuix-HPCchannel: Emmanuel Agullo,Florent Pruvost, case. We work not only on deployment issues, but also upstream—ensuring Gilles Marait, Nathalie Furmento, Marek Felsöci,ˇ Adrien Guilbaud, source code is archived at Software Heritage—and downstream—devising Philippe Swartvagher, Matthieu Simonin) tools and workflows for scientists to use. • Max Delbrück Center for Molecular Medicine (MDC): 2 person-years (Ricardo Wurmus and Mad˘ alin˘ Ionel Patra¸scu) The Guix Workflow Language • Tourbillion Technology: 0.8 person-year (Paul Garlick) The Guix Workflow Language13 (or GWL) is a scientific computing • Université de Paris: 0.5 person-year (Simon Tournier) extension to GNU Guix’s declarative language for package management. It • University of Tennessee Health Science Center (UTHSC): 1.0 person- allows for the declaration of scientific workflows, which will always run in year (Efraim Flashner, Bonface Munyoki, Erik Garrison and Pjotr reproducible environments that GNU Guix automatically prepares. In the Prins) past year theGWL hasseen further developmentto refinethelanguageand • Utrecht Bioinformatics Center (UBC): 0.1 person-year (Roel Janssen) its integration with the Guix environment management features. The GWL was always intended to be used as an extension to Guix, but this presented unique problems. Extensions to Guix may use the Guile modules it provides as a library in addition to accessing Guix package def- initions. With the exact version of Guix under user control via guix pull this put extension developers between a rock and a hard place: they could either dynamically bind to the version of Guix currently in use and hope that theAPIof futureversionsof Guix remainsunchanged compared tothe version they used during development,or they could restrict theextension to a well-known version of Guix with the unfortunate side effect that any reference to packages would be outdated compared to those available in the variant of Guix currently installed by the user. 13 https://workflows.guix.info https://calcul.math.cnrs.fr/2021-01-anf-ust4hpc-2021.html https://hpc.guix.info/blog/2019/10/towards-reproducible-jupyter-notebooks/ 57 17 http://devlog.cnrs.fr/jdev2020/t4 https://lists.gnu.org/archive/html/gwl-devel/2021-02/msg00000.html 56 16 https://guix.gnu.org/manual/en/html_node/Invoking-guix-time_002dmachine.html 15 https://guix.gnu.org/manual/en/html_node/Inferiors.html 14 ihdi n ftemn ulcgalleries. public many the of one in lished uye ol,a scert noewohstidt u oeokpub- notebook a run to tried has who anyone to clear is as world, Jupyter civd n tadesdwa ean eyrlvn su nthe in issue relevant very a remains what addressed it and achieved, n o tm rvl.Ta eyfrtvrindmntae htcnbe can what demonstrated version first very That travel”. “time for and hsJptr“enl ulso uxspotfrrpouil builds reproducible for support Guix on builds “kernel” Jupyter This n esr tsrnigi h xc aesfwr environment. software same exact the in running it’s sure be and u h oeoko n ahn raohr oa rtoyasfo now, from years two or today another, or machine one on notebook the run is-ls nu otecmuaindsrbdi h oeok n can One notebook. the in described computation the to input first-class nesne uxJptrtet otaedpomn n aaa a as data and deployment software treats Guix-Jupyter essence, In bit-reproducible