INRIA Paris/Rocquencourt Conda packages Documentation Release 1.0.0

David Froger

Jun 05, 2017

Contents

1 User documentation 3 1.1 Getting started...... 3 1.2 Package listing...... 4 1.3 Package details...... 9 1.4 Linking with packages...... 10

2 Packager documentation 13 2.1 External links...... 13 2.2 build.sh guidelines...... 13 2.3 Vagrant boxes...... 16 2.4 Package testing...... 17 2.5 Documenting...... 17

3 Contact 19

i ii INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

The INRIA Paris-Rocquencourt SED Conda channel aims at easily distributing scientific softwares and libraries using the Conda package manager. Supported platforms are GNU/Linux 64 bits (old system like CentOS 5 are not supported) and OSX from 10.7 to 10.11. Windows is not supported yet (no plan to support it for now). Root privileges are not required, except for installing compilers in the system. Conda recipes are available.

Contents 1 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

2 Contents CHAPTER 1

User documentation

Getting started

Prerequisites on GNU/Linux

Install gcc, g++, and gfortran. For example, using apt-get:

sudo apt-get install gcc g++ gfortran

or using yum:

yum install gcc-c++ gcc-gfortran

Prerequisites on OSX

Install Xcode Command Line Tools. Download gfortran (select the gfortran-X.X-bin.tar.gz package) from HPC Mac OS X, and install it:

sudo tar xvzf gfortran-X.X-bin.tar.gz -C /

Conda installation

Download the Conda installer, and install conda: On GNU/Linux 64 bits:

wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh bash Miniconda-latest-Linux-x86_64.sh

or, on OSX:

3 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

curl -O https://repo.continuum.io/miniconda/Miniconda-latest-MacOSX-x86_64.sh bash Miniconda-latest-MacOSX-x86_64.sh

You will be asked for a place to install miniconda. Either choose the default $HOME/miniconda or a custom place, for example /local//miniconda. At the end of the installation, you will be asked for modifying your Bash configuration file (~/.bashrc on GNU/Linux, or ~/.bash_profile on OSX): answer yes. To take this modification into account, either open a new terminal, or update the one you are using with:

source ~/.bashrc # on GNU/Linux source ~/.bash_profile # on OSX

Note: the Conda cheat sheet is a usefull ressource to get started with Conda, (managing environments for example is worth to learn).

Installing packages

Install a package provided by the The INRIA Paris-Rocquencourt SED Conda channel , for example:

conda install -c inria-pro-sed petsc

Your may also want to add the inria-pro-sed Conda channel to your configuration (this modify your ~/.condarc):

conda config --add channels inria-pro-sed

in order to omit the -c option:

conda install petsc

Package listing

Linear algebra libraries

Available libraries are:

4 Chapter 1. User documentation INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

Pack- Description Links age METIS Serial Graph Partitioning and Fill-reducing Matrix Ordering recipe, pack- ages Mumps A parallel sparse direct solver recipe, pack- ages Open Open MPI: A high performance message passing library recipe, MPI pack- ages ParMETIS ParMETIS is an MPI-based parallel library that implements a variety of algorithms for recipe, partitioning unstructured graphs, meshes, and for computing fill-reducing orderings of pack- sparse matrices. ages PETSc PETSc: Portable, Extensible Toolkit for Scientific Computation recipe, pack- ages ScaLA- ScaLAPACK is a library of high-performance linear algebra routines for parallel recipe, PACK distributed memory machines. pack- ages Su- SuperLU is a general purpose library for the direct solution of large, sparse, nonsymmetric recipe, perLU_DISTsystems of linear equations on high performance machines. pack- ages Eigen Eigen is a C++ template library for linear algebra. recipe, pack- ages

Note: • Open MPI is build with C, C++ and Fortran API.

Relationships between main libraries are (blue indicate package provided by continuum.io).

1.2. Package listing 5 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

PETSc

SuperLU_DIST Mumps

ScaLAPACK ParMETIS

OpenBLAS OpenMPI METIS

dolfin

boost qt vtk Eigen3 ffc ply OpenMPI

ufl fiat instant

six sympy numpy SWIG CMake

6 Chapter 1. User documentation INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

Image processing libraries (not yet available)

Pack- Description Links age Open- OpenMVS (Multi-View Stereo) is a library for computer-vision scientists and recipe, MVS especially targeted to the Multi-View Stereo reconstruction community. packages Open- “open Multiple View Geometry” is a library for computer-vision scientists and recipe, MVG especially targeted to the Multiple View Geometry community. packages GLFW GLFW is an Open Source, multi-platform library for creating windows with OpenGL recipe, package contexts and receiving input and events. OpenExifOpenExif is an object-oriented library for accessing Exif formatted JPEG image files. recipe, package OSI Osi (Open Solver Interface) provides an abstract base class to a generic linear recipe, package programming (LP) solver, along with derived classes for specific solvers. CLP Clp (Coin-or linear programming) is an open-source linear programming solver recipe, package written in C++. Mesa Mesa is an open-source implementation of the OpenGL specification - a system for recipe not rendering interactive 3D graphics. available, package

OpenMVS

OpenMVG

GLFW OpenEXIF CLP OSI

Mesa

Software engineering tools

Available tools:

1.2. Package listing 7 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

Package Description Links buildbot- Buildbot is an open-source framework for automating software build, test, and recipe, master release processes. packages buildbot- Buildbot is an open-source framework for automating software build, test, and recipe, slave release processes. packages klink A Simple and Clean Sphinx Docs Theme recipe, packages nbtools Tools for managing IPython notebooks recipe, packages Relationships between main libraries are (blue indicate package provided by continuum.io).

Buildbot Master

python2.7 Twisted Jinja2 dateutil SQLAlchemy Migrate

SQLAlchemy Decorator Tempita

nbtools

Python Jinja2 IPython Notebook matplotlib nose ghdiff

chardet

File format libraries

Available libraries:

8 Chapter 1. User documentation INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

Pack- Description Links age GetPot Powerful Input File and Command Line Parser recipe, packages h5utils h5utils is a set of utilities for visualization and conversion of scientific data in the free, recipe, portable HDF5 format. packages HDF5 HDF5 is a data model, library, and file format for storing and managing data. recipe, packages LM5 Reads and writes mesh files (.mesh) or physical solution files (.sol) recipe, packages yaml- A YAML parser and emitter in C++ recipe, cpp packages

Note: • HDF5 is build with C and C++ API.

Python bindings

Available Python bindings are: Pack- Description Links age h5py The h5py package is a Pythonic interface to the HDF5 binary data format. recipe, packages mpi4py This package provides Python bindings for the Message Passing Interface (MPI) recipe, standard. packages petsc4py PETSc for Python recipe, packages

Package details

Integer and real types

Numerical algebra libraries make use of 32 bits index integers and 64 bits floating point numbers.

Note: OpenBLAS is provided by continuum analytics. The package openblas-0.2.14-3 contains the file in- clude/openblas_config.h. This file does not defined OPENBLAS_USE64BITINT, so blasint is a int (32 bits).

C++ standard library on OSX

On OSX, two implementations of the C++ standard library are available: • The GNU implementation: libstdc++.dylib • The Clang implementation: libc++.dylib The ABI of these two libraries are incompatibles. All packages in the INRIA Paris-Rocquencourt SED Conda channel are built with the Clang implementation.

1.3. Package details 9 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

Note: A Python script (for example), may import a C++ extension linked with libc++.dylib and another C++ exten- sion linked with libstdc++.dylib without problems, as long as the two extensions do not directly exchange C++ data structures.

glibc and gcc versions

Packages are build on Centos 6.6, using glibc version 2.12, and gcc version 4.4.7. As a consequence, packages will work on all GNU/Linux distribution providing glibc version 2.12 or greater. Taken from distrowatch, here are some examples: distribution glibc gcc centos 5.11 2.5 4.1.2 centos 6.6 2.12 4.4.7 centos 7.0 2.17 4.8.3 ubuntu 12.04 2.15 4.6.3 ubuntu 14.04 2.19 4.8.2 debian 7 2.13 4.7.2 fedora 17 2.15 4.7.0 suse 12.1 2.12.1 4.6.2 Instead of using the system gcc, we may in the future use the gcc conda package, which provide a recent gcc version, linked with an old glibc version. But currently (5 October, 2015), gcc Conda package (4.8.5-3, in asmeurer channel), has too much problems: • https://groups.google.com/a/continuum.io/forum/#!search/froger/conda/Lb_Vnz6O85s/20dsvlvlBwAJ • https://github.com/conda/conda-recipes/issues/435 • https://github.com/conda/conda-recipes/issues/431

Build environment

Packages are built on Vagrant box, and hosted on anaconda in channels tagged with the box name and version.

Linking with packages

Finding path to conda environment

The $CONDA_ENV_PATH environment variable is defined when an environment is activated. If no environment is activate, the default environment is used, but $CONDA_ENV_PATH is not defined. A way to compute Conda environment path is:

CONDA_ENV_PATH= $(conda info -e | grep' *' | tr -s'' | cut -d'' -f3 )

10 Chapter 1. User documentation INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

Environment variables

Warning: Beware of using environment variables like LD_LIBRARY_PATH or DYLD_LIBRARY_PATH. They are not required by conda packages, and wrong values can lead to errors.

Compilers

Or GNU/Linux, use gcc, g++ or gfortran compiler. On OSX, use clang, clang++ or gfortran compiler.

Warning: Depending on your OSX version, gcc is or is not an alias to clang.

C++ standard library

Warning: On OSX, always build and link with the -stdlib=libc++ flags, see C++ standard library on OSX.

Header and library path

When compiling, provided path to header and library paths as usual:

-L$CONDA_ENV_PATH/lib -I$CONDA_ENV_PATH/include

When linking on OSX (not required for GNU/Linux), also set the rpath:

-L$CONDA_ENV_PATH/lib -Wl,-rpath,$CONDA_ENV_PATH/lib

Note: When using mpicc or mpic++, -L and -I flags are automatically set.

1.4. Linking with packages 11 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

12 Chapter 1. User documentation CHAPTER 2

Packager documentation

External links

External links are summarized here:

GitHub

• Conda recipes • CentOS 6 box config • CentOS 7 box config • OSX box config • Testing scripts

Anaconda cloud

• Conda channel on Anaconda Cloud

HashiCorp

• Vagrant boxes. build.sh guidelines.

Building packages is documented in the Conda documentation. Conda recipes are available on GitHub.

13 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

Shared libraries

Build shared libraries when possible (requires -fPIC flag). If building a static library, use the -fPIC flag too: the static library may be used latter in a dynamic library.

Plateform conditional code

If different logic must be performed on GNU/Linux and OSX, use the construct:

if [" $(uname)"== "Darwin"] then # OSX recipe else # GNU/Linux recipe fi

Verbose build

Prefer verbose build, to facilitate debugging. For example with autotools:

makeV=1

or with CMake:

VERBOSE=1 make

clang and libc++.dylib

When building C code on OSX: • use clang or mpicc • MACOSX_DEPLOYMENT_TARGET is already automatically set to 10.6 by conda-build. When building C++ code on OSX: • Use clang++ or mpicxx. • Build for 10.7 and upper. • Link with libc++.dylib For example, for typical autotools based software:

export MACOSX_DEPLOYMENT_TARGET=10.7

exportCC=clang export CXX=clang++ export CFLAGS="-mmacosx-version-min=10.7" export CXXFLAGS="-stdlib=libc++ -mmacosx-version-min=10.7" export LDFLAGS="-stdlib=libc++ -mmacosx-version-min=10.7"

./configure --prefix=$PREFIX makeV=1

14 Chapter 2. Packager documentation INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

Warning: When building Python extension, CFLAGS or LDFLAGS environment variables should not be set (see next section).

or for CMake based software: export MACOSX_DEPLOYMENT_TARGET=10.7

VERBOSE=1 cmake \ -DCMAKE_C_COMPILER=clang \ -DCMAKE_C_FLAGS="-mmacosx-version-min=10.7" \ -DCMAKE_CXX_COMPILER=clang++ \ -DCMAKE_CXX_FLAGS="-stdlib=libc++ -mmacosx-version-min=10.7" \ -DCMAKE_SHARED_LINKER_FLAGS="-stdlib=libc++ -mmacosx-version-min=10.7" \ -DCMAKE_EXE_LINKER_FLAGS="-stdlib=libc++ -mmacosx-version-min=10.7" \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ ..

Python extension

Build tools call the distutils standard python module (or the python-config command line tool) to set compiler flags for building python extensions. Python (provided by Conda) has been built with an old XCode version, and distutils (or python-config) may (see bellow) return the flag: -isysroot /Developer/SDKs/MacOSX10.5.sdk

However, SDK are now located in: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs

If /Developer/SDKs/MacOSX10.5.sdk is not found (which is the case for the conda-build-oxs-10.9 vagrant box), dis- tutils (or python-config) remove the -isysroot flag except if the corresponding environment variables (CFLAGS or LDFLAGS) are set. See the code in the ditutils module. -isysroot is not required because MACOSX_DEPLOYMENT_TARGET=10.7 and -mmacosx-version-min=10.7 are set, so the better is to not set CFLAGS, LDFLAGS (and CXXFLAGS for being consistant), and use build tool command line instead, for example, in build.sh: export MACOSX_DEPLOYMENT_TARGET=10.7

exportCC=clang export CXX=clang++

unset CXXFLAGS unset CFLAGS unset LDFLAGS

$PYTHON waf configure \ --prefix=$PREFIX \ --cflags='-O3 -DNDEBUG -mmacosx-version-min=10.7 -arch x86_64' \ --cxxflags='-O3 -DNDEBUG -stdlib=libc++ -mmacosx-version-min=10.7 -arch x86_64' \ --ldflags="-L$PREFIX/lib -O3 -DNDEBUG -stdlib=libc++ -mmacosx-version-min=10.7 -

˓→arch x86_64"

2.2. build.sh guidelines. 15 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

Path to conda provided libraries on OSX

On OS X, path to libraries provided by Conda must be provided with -L$PREFIX/lib.

Vagrant boxes

Packages are built on Vagrant boxes for reproducibility. Make sure to have the most recent version of VirtualBox and Vagrant. Vagrant boxes are hosted on hashicorp.

Using vagrant boxes

This section contains instructions to use conda-build-centos-6.6 box, the same apply for conda-build-centos-7.0 or conda-build-oxs-10.9 boxes. Create a directory and initialize the box in it:

mkdir -p inria-pro-sed/conda-build-centos-6.6 cd inria-pro-sed/conda-build-centos-6.6 vagrant init inria-pro-sed/conda-build-centos-6.6

In the Vagrantfile, synchronise your Conda recipe directory:

config.vm.synced_folder"/path/to/repo/sed-pro-inria/conda-recipes", "/home/vagrant/repo/sed-pro-inria/conda-recipes"

and increase cpu and memory:

# Memory and CPU. config.vm.provider "virtualbox" do |v| v.memory= 4096 v.cpus=8 end

Start the machine, and connect to it:

vagrant up vagrant ssh

Build your Conda package and use the upload alias to upload on anaconda.org: • on the inria-pro-sed user, • on the main,boxname-boxversion channels.

Build a new box

Make changes in Vangrantfile, ressources/ or script/. Update CHANGELOG. Update version number in ressources/_bashrc. Between two tagged versions, set the version number to X.X.Xdev in ressources/_bashrc.

16 Chapter 2. Packager documentation INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

Commit and tag: git add ressources/_bashrc # and other files git commit -m "Bump conda-build-centos-6.6 version number to X.X.X" git tag centos-6.6-X.X.X

Destroy old box, and create a new one: vagrant destroy vagrant up

Package the box, and add into Vagrant: vagrant package --output conda-build-centos-6.6.box

Upload the box to using the web interface. Or generate a token from , store it in $ATLAS_TOKEN and use the API. For example:

# TODO: create a provider

# Upload a box for provider curl https://atlas.hashicorp.com/api/v1/box/inria-pro-sed/conda-build-centos-6.6/

˓→version/$VAGRANT_BOX_VERSION/provider/virtualbox/upload?access_token=$ATLAS_TOKEN UPLOAD_PATH=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX # Use value returned by previous

˓→command curl -X PUT --upload-file $AGRANT_BOX_NAME.box https://binstore.hashicorp.com/$UPLOAD_

˓→PATH # display nothing, just wait...

Package testing

Some scripts exist to test conda packages on various operating systems.

Documenting

Install graphviz from apt-get or other. Install sphinx: conda install python=3.5 sphinx

2.4. Package testing 17 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0

18 Chapter 2. Packager documentation CHAPTER 3

Contact

Any feedback is welcome! Open a issue or send a pull request on GitHub, or contact me at [email protected].

19