Compiling and installing software, manually or automatically, file comparison J. M. P. Alves Laboratory of Genomics & Bioinformatics in Parasitology Department of Parasitology, ICB, USP Software for all ● Operating systems nowadays, differently from the old times, come with a large selection of user applications installed by default ● Those are programs that we use most of the time: office software, web browsers, image editors, music players, games, etc. etc. etc. ● Linux distros such as Ubuntu and Debian include most of the commonly used programs in their default download image ● However, the universe of software is huge, and very diverse; it is not possible to include even a small sample of every kind of application that exists out there ● That is where being able to install other programs comes in handy J.M.P. Alves 2 / 55 BMP0260 / ICB5765 / IBI5765 Software for all ● Until the 1990s, the Internet was not as available for the general public as it is today, so software had to be distributed or copied in tape or disks (magnetic, like floppy disks, or optical, like CDs), or even printed 5.25” floppy, 3.5” floppy, 8” floppy, 175 kB to 1.2 MB 360 or 720 kB 1.44 MB J.M.P. Alves 3 / 55 BMP0260 / ICB5765 / IBI5765 40 years of external storage By avaragado, at http://flickr.com/photos/89394041@N00/6960433672 J.M.P. Alves 4 / 55 BMP0260 / ICB5765 / IBI5765 The modern era ● With the Web, anyone with an Internet connection became able to find software ● Certain specialized collections of software, called repositories, have been set up and serve as centralized sources ● Classical example in biological software, the IUBio Software section (since 1989!): http://iubio.bio.indiana.edu/software/ ● Many of these repositories are dedicated to a certain kind of software, e.g. from a platform (CRAN for R, CPAN for Perl etc.) ● One can use a specific tool to get new software from a repository, or alternatively just use a Web browser – it depends on each repository ● Repositories can add safety: tested and examined by the maintainers J.M.P. Alves 5 / 55 BMP0260 / ICB5765 / IBI5765 The Linux contribution ● It is very often the case that a new program that you want to run depends on some other piece(s) of software ● These dependencies may be other independent programs (which might do something that your new program needs) or shared software libraries (which are collections of code that do not operate independently, but just “help” other programs) ● In either case, we can only use our new program if the dependencies are installed ● And the dependencies themselves might (and usually do) depend on other programs or libraries… ● As you’ve probably noticed by now, this can be quite a problem J.M.P. Alves 6 / 55 BMP0260 / ICB5765 / IBI5765 Dependency hell ● To solve this problem, a system of package management has been established, around 1997 to 1998, in different Linux distributions ● A package is a program plus all the other accessory files it needs (like eventual data files, man pages, specific libraries etc.) ● One of the creators of Debian GNU/Linux, Ian Murdock, has said: “What’s the single biggest advancement Linux has brought to the industry? Package Management –or, more specifically, the ability to install and upgrade software over the network in a seamlessly integrated fashion” ● Nowadays, everyone knows something similar to Linux repositories and package management: the app stores ● Managing a package means: install, remove, update – the package itself and all of its dependencies J.M.P. Alves 7 / 55 BMP0260 / ICB5765 / IBI5765 Managing packages ● Package management also ensures that all versions of the dependencies involved are compatible ● The program in a package is compiled and ready to run ● In the early days of Linux (remember, the kernel was written in 1991), two distros came up with their own way of packaging programs and then managing the resulting packages ● Those pioneers (and dominating to this day) were: ● RedHat, 1997, with the RPM system (RedHat Package Manager) ● Debian, 1998, with the DEB system ● Debian (which is the system installed on the remote server) is the direct ancestor of Ubuntu, one of the most used distros nowadays, so we will focus on package management using the DEB system J.M.P. Alves 8 / 55 BMP0260 / ICB5765 / IBI5765 Taking a look at the GUI ● Every big Linux distro has now its own repository, with many gigabytes of all kinds of software; for example, see: Debian packages: https://www.debian.org/distrib/packages Ubuntu packages: https://packages.ubuntu.com/focal/ ● In a desktop Ubuntu machine, you can use the Ubuntu Software program ● This program is the default way, in Ubuntu, for exploring the repository and managing packages – I myself only use it when the system is brand new, to install Synaptic (a much better package manager in my opinion… but that depends on taste) ● In Ubuntu Software you can see what is installed, as well as what is available for installation, which is a mouse click away ● You cannot install anything without an administrator password J.M.P. Alves 9 / 55 BMP0260 / ICB5765 / IBI5765 Back to the command-line ● Everything that can be done in the graphical apps can also be done in the nurturing and friendly environment of the CLI, of course ● The Debian family of distros all use the DEB file format ● The most popular CLI program for DEB package management is the APT system ● The system maintains a list of installed packages, as well as those available in the repositories registered (which can be several) ● The first thing to do is to update this information apt-get update ● This command will download all the latest software lists from the repositories J.M.P. Alves 10 / 55 BMP0260 / ICB5765 / IBI5765 Searching for packages ● There are tens of thousands of packages out there, therefore, searching is an important task apt-cache search regex ● The regular expression can be just simple words (e.g., likelihood), as always, and it will search the package names and descriptions ● Now that you have found some packages that look like they might be what you are looking for, its time to ask for more information apt-cache show ncbi-blast+ apt-cache showpkg ncbi-blast+ ● These two commands will show basic information about the package, as well as more detailed information such as dependencies J.M.P. Alves 11 / 55 BMP0260 / ICB5765 / IBI5765 Installing packages ● Having decided that a package does what you need, now it is time to install it (as root or, in Ubuntu, using sudo if you have admin powers) apt-get install raxml sudo apt-get install raxml (not available in Debian) ● The first command will fail if you are not root ● The second command uses a program, sudo, which allows one to run any command as another user (by default, root), using one’s own password ● Debian is very security-conscious, so it does not have sudo installed by default; Ubuntu, on the other hand, does not have a root user J.M.P. Alves 12 / 55 BMP0260 / ICB5765 / IBI5765 Upgrading packages ● Another reason why package management is very useful is that it deals with updates as well apt-get upgrade raxml apt-get upgrade ● As seen, one can give the name of the package to be updated, of just ask for all updates (second command) ● If the package that is installed has no newer version available, the user is told so; e.g.: muscle is already the newest version (1:3.8.31+dfsg-1). ● If one tries to upgrade a package that was not installed, apt-get tries to install it, after confirmation from the user: The following NEW packages will be installed: J.M.P. Alves 13 / 55 BMP0260 / ICB5765 / IBI5765 Removing packages ● Maybe that package whose description looked good does not really do what you wanted… ● Time to remove it: apt-get remove raxml apt-get purge raxml ● The second option is more drastic: it removes any specific configuration files as well as the program itself ● Leaving the configuration files could be useful in the future, specially if the system was using tailored configurations, different from the default J.M.P. Alves 14 / 55 BMP0260 / ICB5765 / IBI5765 Out of repository? ● Although the repositories have a lot of stuff, they are far from having everything ● Also, it is often the case that the distro’s repository has a rather old version of the program – that is a problem specially for Debian, which is a distro that cares a lot about software security and stability ● However, package files can be created by anyone, not just repository maintainers ● Many projects distribute DEB and/or RPM files so you can install them directly; e.g.: http://donate.libreoffice.org/pt/dl/deb-x86_64/5.3.3/pt-BR/ LibreOffice_5.3.3_Linux_x86-64_deb.tar.gz J.M.P. Alves 15 / 55 BMP0260 / ICB5765 / IBI5765 Out of repository? ● The greatest disadvantage of directly installing a package file is that there won’t be any automatic dependency resolution ● Nonetheless, the system will still check if dependencies are all available and are the correct versions – and refuse to install if not ● A more basic, “low-level” tool is used for directly installing packages: dpkg -i *.deb ● This command will install (upgrading, if there is an older version present) all DEB files present in the current directory ● The dpkg program is used by APT tools, by the way, so when you are using apt-get, you are indirectly using the less friendly dpkg J.M.P. Alves 16 / 55 BMP0260 / ICB5765 / IBI5765 What else? ● As you saw, installing programs in Linux is a breeze when they are in the distro’s repository ● When they are not, but there is an available package, things are still quite easy as long as there are no dependency issues: the program is already compiled and in an easy-to-install form ● What if there is neither? J.M.P.