Open Is Not Enough: Benefits from Debian As an Integrated

Open is not enough: beneﬁts from Debian as an integrated, community-driven computing platform

Yaroslav O. Halchenko & Michael Hanke

Debian Project Dartmouth College, New Hampshire University of Magdeburg, Germany

SEA 2013, Boulder CO

March 2 2013

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 1 / 38 Am I the right person?

To: [email protected] Subject: Duplicating Debian Installations? From: [email protected] (Tres Hofmeister) Date: Mon, 19 Aug 1996 19:10:41 -0600 (MDT)

Can someone point me towards info. on how one can duplicate the packages installed on one Debian system on another? Going through dselect by hand seems just a bit too tedious for multiple installations... Thanks.

-- Tres Hofmeister Research Applications Program [email protected] National Center for Atmospheric Research

https://lists.debian.org/debian-user/1996/08/msg00890.html H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 2 / 38 Based on a true story...

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 3 / 38 Who is talking?

I and

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 4 / 38 Who is talking?

http://www.pymvpa.org H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 5 / 38 Who is talking?

http://www.debian.org H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 5 / 38 Who is talking?

http://neuro.debian.net H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 5 / 38 Our domain

Poline et. al, Data sharing in neuroimaging research, 10.3389/fninf.2012.00009 H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 6 / 38 Our belief #1

There is no alternative to free and open-source software for scientiﬁc research.

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 7 / 38 “Standard” model of scientiﬁc software deployment To download and install A on your system, you need to: Register/Request an account from the B Read these notes for C and D systems Download the source code and/or appropriate binaries for your system from E Remove old versions Install software Test software

Common problem: Availability != Accessibility

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 8 / 38 Common problem: Availability != Accessibility

“Standard” model of scientiﬁc software deployment To download and install A on your system, you need to: Register/Request an account from the B Read these notes for C and D systems Download the source code and/or appropriate binaries for your system from E Remove old versions Install software Test software

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 8 / 38 Common problem: Availability != Accessibility

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 8 / 38 Student’s life is short: Having A on one box should be enough I better not have to upgrade it ever again I will not even think about F and G alternatives I value others’ time: let me blog a “HOWTO install A v. x.y.z on C”

Positive

Only an hour and “A” seems to be running – I can get back to do research again!

Possible user experiences

Not so positive

Software developers have no access to my system X and I need to build a few (dozens) of additional pre-requisites ﬁrst: I better ask our IT personnel to deploy it – usually takes less than a month PhD students have all the time in the universe . . .

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 9 / 38 Student’s life is short: Having A on one box should be enough I better not have to upgrade it ever again I will not even think about F and G alternatives I value others’ time: let me blog a “HOWTO install A v. x.y.z on C”

Possible user experiences

Not so positive

Positive

Only an hour and “A” seems to be running – I can get back to do research again!

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 9 / 38 Possible user experiences

Not so positive

Positive

Only an hour and “A” seems to be running – I can get back to do research again! Student’s life is short: Having A on one box should be enough I better not have to upgrade it ever again I will not even think about F and G alternatives I value others’ time: let me blog a “HOWTO install A v. x.y.z on C”

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 9 / 38 Hanke, M. (2012). Share your tools! But fear the wombat! http://youtu.be/8t6znEOEDVo Not so positive

Positive

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 9 / 38 Scientists report:

Hanke M and Halchenko YO (2011) Neuroscience runs on GNU/Linux. Front. Neuroinform. 5:8. doi: 10.3389/fninf.2011.00008 GNU/Linux’es MCC Interim 1992 1992 TAMU SLS GNU/Linux Distribution Timeline Yggdrasil DLD

1993 Version 12.2 1993 Slackware Debian A. Lundqvist, D. Rodic - futurist.se/gldt LST

Bogus Published under the GNU Free Documentation License S.u.S.E 1994 1994 Red Hat WGS Linux Pro Linux-FT Xdenu MNIS Mini Jurix Trans-Ameritech 1995 1995 Unifix Linux Universe Craftworks Caldera 1996 1996 DILINUX mkLinux Monkey 1997 1997 LinuxPPC Burapha UltraPenguin Weaver DOSLINUX Conectiva Stampede Buhawi PLD Eurielec µClinux 1998 1998 Turbolinux Mandrake ROCK Fermi Demolinux Project Ballantin Project tomsrtbt SuSE Coyote e-smith KRUD eIT easyLinux Eridani Yellow Dog 1999 1999 Vine FREESCO Red Flag Corel Armed Immunix Kondara Libranet Peanut Scratch Linux From Enoch BluePoint Storm Astaro Linpus ELX (Arma) Omoikane Trustix 2000 2000 Finnix Tiny BU VectorLinux Miracle Gibraltar Happy SmoothWall GPL Knoppix Pingo Caixa Mágica Ututo Best LEAF Rocks ClarkConnect SLES CRUX Beehive 2001 2001 Midori ASP ALT Leka Rescue Floppy Leka Rescue Progeny SuperRescue Openwall Independence dyne:bolic OpenWRT K12 Skolelinux DeMuDi SME Server SME HP Secure Ark Lindows Xandros NetStation Virtual Sorcerer Aurora EnGarde Redmond gnuLiNex IPCop Haydar Tinfoil Hat Lunar Arch 2002 2002 Red Hat Enterprise Gentoo Source Mage Source SOT EvilEntity Slax Bayanihan SCO Lycoris Topologilinux LinuxConsole BLAG DeLi Aurox United Linux Rxart GoboLinux LinuxInstall Yoper ZENIX Kurumin Gentoox Quantian 2003 2003 miniwoody Morphix Berry Jollix UHU MEPIS GeeXboX Puppy Linux Damn Small AnNyung AL-AMLUG KnoppMyth ATmission Thinstation Stux SAM PCLinuxOS ASLinux SUSE Devil Kwort SmoothWall Express Impi Fedora Core Fedora Danix Sun JDS NST CentOS blackPanther Feather White Box OpenSLS Shabdix Kanotix Bonzai Nitix Linspire Dizinha Bluewall STUX BioBrew DARKSTAR DNALinux Kalango Amber K-DEMar Bioknoppix Guadalinex Hiwix AUSTRUMI Ututo-e 2004 2004 Euronode QiLinux Natures Linux Joatha DeadCD LBA CERN Openfiler Aquamorph Live Clonezilla Scientific Minislack Linux Security Auditor MAX Asianux Navyn OS ZoneCD B2D Desktop Underground Octoz GoblinX Kaella Specifix HostGIS Hiweed Whoppix Momonga StartCom SimplyMEPIS grml Annvix Hikarunix FoX T2 VidaLinux Kate NimbleX Ubuntu Frugalware MOPSLinux NLD Asterisk@Home BeatriX Tao Arco-Debian Paldo MoLinux Slamd64 Parsix Resulinux Pentoo Tirwal BlankOn Voyage aLinux Foresight epiOS Arabian Poseidon 2005 2005 Archie Mandriva 64Studio Trisquel Kubuntu ParallelKnoppix KateOS NetSecL Symphony OS Symphony rPath Ututo XS MythDora VENENUX LliureX WHAX AsianLinux pQui Elive Wolvix IPFire Zenwalk VLOS Lihuen Tuquito Musix Hedinux SLAMPP Slackintosh RR4 BrazilFW Edubuntu OS2005 ThinClientOS Bintoo Kororaa nUbuntu NepaLinux Pardus Sauver Jarro Negro Jarro Dreamlinux Epidemic Vyatta 2006 2006 Ophcrack Librassoc Olive Freespire Alpine NeoDizinha RIP DSL-N Backtrack trixbox SLED Voltalinux Gnu-HALO Xubuntu Everest Zeroshell Bardinux Christian Edition Christian PUD PureOS Sabayon Ekaaty Mint Linkat Bluewhite64 How-Tux Absolute Oracle Enterprise TinyMe Papug Damn Vulnerable Linux gNewSense sidux openSUSE Ulteo Satanic Edition Wifislax Elastix Granular Oz DEFT BOSS Muslim Edition Muslim Uberyl TEENpup Patinho Faminto Patinho 2007 2007 Madbox LPS SliTaz Webconverger Fedora openmamba Mythbuntu Calculate Moblin ComFusion SMS antiX Ubuntulite Geubuntu Greenie Vixta Syllable Server Syllable Kiwi Fluxbuntu gOS Mamona NuTyX Toorox Eeebuntu PLoP CrunchBang Zebuntu SystemRescueCD PelicanHPC Ubuntu Rescue Remix Ubuntu Rescue 2008 2008 GALPon MiniNo GALPon Ultimate Edition Exherbo Simplis xPUD OpenGEU Estrella Roja Estrella Ubuntu eee Macpup Inquisitor MoonOS Maryan Ojuba Moblin 2 Moblin Funtoo Zorin OS Zorin eBox U-lite XBMC Live XBMC Newtoos Arc-Live VortexBox Chaox Wifiway Metamorphose ZevenOS Chakra PLATYPUX Xange BlueOnyx Tiny Core LinHES Kongoni A/V Nova Qimo 2009 2009 Dragora Linvo Lin2Go Lubuntu Easy Peasy Qomo Sabily kademar FireFly webOS Canaima AV Tails wattOS KahelOS OS Chromium CTKarchLive Arch Linux Arch Linux ARM Salix ClearOS Leeenux Maemo Hanthana Deepin Semplice Unity Ylmf ArchBang Element Igelle 0 2010 2010 Netrunner QubesOS AgiliaLinux GnackTrack OS Peppermint Dax OS Quirky MeeGo OS Aurora Superb Mini Server Mini Superb Tiny SliTaz DoudouLinux Synergy Imagineos Jolicloud FrameOS ChameleonOS AriOS Legacy OS Zentyal aptosid Linux Mint Debian Linux Mint Dream Studio Dream Mageia Pinguy OS Pinguy 4M Fusion Parabola Fuduntu ConnochaetOS Swift Bodhi Progress Parsidora GNUGuitarINUX SalineOS Conducit Liquid Lemur Liquid 2011 2011 CTKArch Garuda AnikOS SlaXBMC Meego Harmattan Meego Porteus Manjaro Pear OS Pear Viperr 2012 2012

http://futurist.se/gldt H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 11 / 38 GNU/Linux’es MCC Interim 1992 1992 TAMU SLS GNU/Linux Distribution Timeline Yggdrasil DLD

1993 Version 12.2 1993 Slackware Debian A. Lundqvist, D. Rodic - futurist.se/gldt LST

http://futurist.se/gldt H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 11 / 38 Our belief #2

Debian provides the best proven, scalable, and sustainable approach toward free and open-source software platform for scientiﬁc research.

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 12 / 38 Extensible distribution and archival infrastructure

Debian’s recipe

Open standards Vibrant community: democracy and do-o-cracy Integration starting with the core OS Binary distribution based on source packages

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 13 / 38 Extensible distribution and archival infrastructure

Debian’s recipe: Binary distribution

Open standards Vibrant community: democracy and do-o-cracy Integration starting with the core OS Binary distribution based on source packages

http://xkcd.com/303 H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 13 / 38 Debian’s recipe

Open standards Vibrant community: democracy and do-o-cracy Integration starting with the core OS Binary distribution based on source packages Extensible distribution and archival infrastructure

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 13 / 38 Pros

you know that everything is free and open-source eases integration helps to assure longevity http://popcon.debian.org is there to replace “registration” pages

Legal assurance Was: Register/Request an account from the B

3 Debian archive areas

main – software under DFSG compliant licenses 1 contrib – DFSG-wannabees (depend on non-free) or helpers (e.g., matlab-support) non-free – (re-)distributable but restricted

1See Debian Social Contract http://www.debian.org/social_contract H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 14 / 38 Legal assurance Was: Register/Request an account from the B

3 Debian archive areas

main – software under DFSG compliant licenses 1 contrib – DFSG-wannabees (depend on non-free) or helpers (e.g., matlab-support) non-free – (re-)distributable but restricted

Pros

you know that everything is free and open-source eases integration helps to assure longevity http://popcon.debian.org is there to replace “registration” pages

1See Debian Social Contract http://www.debian.org/social_contract H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 14 / 38 Legal assurance Was: Register/Request an account from the B

3 Debian archive areas

main – software under DFSG compliant licenses 1 contrib – DFSG-wannabees (depend on non-free) or helpers (e.g., matlab-support) non-free – (re-)distributable but restricted

Pros

you know that everything is free and open-source eases integration helps to assure longevity http://popcon.debian.org is there to replace “registration” pages

1See Debian Social Contract http://www.debian.org/social_contract H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 14 / 38 Legal assurance Was: Register/Request an account from the B

Recommendations

use standard OSI (http://opensource.org) and DFSG compliant licenses do not restrict the domain of application without need do not insert code/data with questionable license terms maintain a top-level list of licenses used in you project beware-of and/or state your trademark policies

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 14 / 38 Building

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 15 / 38 Building : developers

Image by Lewis Hine, 1930

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 15 / 38 Building : well-engineered software

Image by Lewis Hine, 1930 Image by Ad Meskens H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 15 / 38 Building : “state-of-art”

anyone can build any PACKAGE: apt-get source -b PACKAGE simpliﬁes contributions allow for archive-wide QA rebuilds library migrations new compilers new ports

Building Was: If you decide to go with building from source code, good luck!

Debian packaging strictly follows Debian Policy1 3rd-party modules become 1st-class citizens Figure out dependencies once – specify in debian/control Figure out building/installation once – specify in debian/rules

1http://www.debian.org/doc/debian-policy/ H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 16 / 38 Building Was: If you decide to go with building from source code, good luck!

Pros

anyone can build any PACKAGE: apt-get source -b PACKAGE simpliﬁes contributions allow for archive-wide QA rebuilds library migrations new compilers new ports

1http://www.debian.org/doc/debian-policy/ H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 16 / 38 Above recommendations will help to deliver your software to any integration/distribution platform, not just Debian.

Building Was: If you decide to go with building from source code, good luck!

Recommendations keep source distribution modular: code-vs-data keep 3rd party as 3rd party – do not clone have a deterministic version provide exhaustive speciﬁcation of (build-)dependencies assure API/ABI compatibility if you deliver libraries: http://upstream-tracker.org/ http://github.com/lvc/abi-compliance-checker

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 16 / 38 Building Was: If you decide to go with building from source code, good luck!

Above recommendations will help to deliver your software to any integration/distribution platform, not just Debian.

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 16 / 38 Testing

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 17 / 38 Bruno “btco” Oliveira https://plus.google.com/+BrunoOliveira/posts/MGxauXypb1Y Pros

obtain a CI farm with > 13 architectures/ports have I mentioned archive wide QA rebuilds? Go ahead of the OS not behind preclude problems on users’ deployments

Testing Was: Test software

debian/rules: can exercise tests while building the binary packages

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 18 / 38 Testing Was: Test software

debian/rules: can exercise tests while building the binary packages

Pros

obtain a CI farm with > 13 architectures/ports have I mentioned archive wide QA rebuilds? Go ahead of the OS not behind preclude problems on users’ deployments

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 18 / 38 Testing Was: Test software

debian/rules: can exercise tests while building the binary packages

Pros

obtain a CI farm with > 13 architectures/ports have I mentioned archive wide QA rebuilds? Go ahead of the OS not behind preclude problems on users’ deployments

autopkgtest: automatic as-installed package testing

Pros

exercise testing on installed machines could be “heavy” tests

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 18 / 38 more problems we catch before delivering to users – less of WOMBAT

Testing Was: Test software

Recommendations

share your unit-, regression-, integration- tests work on your tests coverage

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 19 / 38 Testing Was: Test software

Recommendations

share your unit-, regression-, integration- tests work on your tests coverage more problems we catch before delivering to users – less of WOMBAT

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 19 / 38 Install workload management system apt-get install condor

Install the entire suite of packages for meteorology apt-get install science-meteorology

Keep the whole system up-to-date apt-get update && apt-get upgrade

Installation, upgrades Was: Remove old versions; Install software

Install simple editor apt-get install gedit

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 20 / 38 Install the entire suite of packages for meteorology apt-get install science-meteorology

Keep the whole system up-to-date apt-get update && apt-get upgrade

Installation, upgrades Was: Remove old versions; Install software

Install simple editor apt-get install gedit

Install workload management system apt-get install condor

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 20 / 38 Keep the whole system up-to-date apt-get update && apt-get upgrade

Installation, upgrades Was: Remove old versions; Install software

Install simple editor apt-get install gedit

Install workload management system apt-get install condor

Install the entire suite of packages for meteorology apt-get install science-meteorology

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 20 / 38 Installation, upgrades Was: Remove old versions; Install software

Install simple editor apt-get install gedit

Install workload management system apt-get install condor

Install the entire suite of packages for meteorology apt-get install science-meteorology

Keep the whole system up-to-date apt-get update && apt-get upgrade

H2 (Debian/Dartmouth+Magdeburg) Open is not enough SEA 2013 20 / 38 Debian GIS: http://wiki.debian.org/DebianGis http://blends.alioth.debian.org/gis/tasks/ Francesco P. Lovergine”