Porting legacy software packages to the Conda Package Manager Joe Asercion, Fermi Science Support Center, NASA/GSFC Previous Process FSSC Previous Process FSSC Release Tag Previous Process FSSC Release Tag Code Ingestion Previous Process FSSC Release Tag Code Ingestion Builds on supported systems Previous Process FSSC Release Tag Code Ingestion Push back Builds on changes supported systems Testing Previous Process FSSC Release Tag Code Ingestion Push back Builds on changes supported systems Testing Packaging Previous Process FSSC Release Release Tag Code Ingestion All binaries and Push back Builds on source made changes supported systems available on the FSSC’s website Testing Packaging Previous Process Issues • Very long development cycle • Difficult dependency • Many bottlenecks management • Increase in build instability • Frequent library collision errors with user machines • Duplication of effort • Large number of individual • Large download size binaries to support Goals of Process Overhaul • Continuous Integration/Release Model • Faster report/patch release cycle • Increased stability in the long term • Increased Automation • Increase process efficiency • Increased Process Transparency • Improved user experience • Better dependency management Conda Package Manager • Languages: Python, Ruby, R, C/C++, Lua, Scala, Java, JavaScript • Combinable with industry standard CI systems • Developed and maintained by Anaconda (a.k.a. Continuum Analytics) • Variety of channels hosting downloadable packages Packaging with Conda Conda Build Meta.yaml Build.sh/bld.bat • Contains metadata of the package to • Executed during the build stage be build • Written like standard build script • Contains metadata needed for build • Dependencies, system requirements, etc • Ideally minimalist • Allows for staged environment • Any customization not handled by specificity meta.yaml can be implemented here. • Large amount of customization available • Jinja2 macros Meta.yaml Custom Variables Package versioning metadata Source code handling Run stage Build process parameters dependencies Build stage dependencies Host stage dependencies Test handling Defining Package Metadata Only mandatory sections: package/name & package version Package Source Build Descriptive metadata for the Conda can pull from multiple Specifies metadata which describes package. Name and version string sources natively. the build itself including build must be included. targets • url Build number is used to specify new • Local path builds of the same version • Git • Hg • Svn Conda can handle aggregate builds from multiple source locations Conda Build Environments Build Host Run • Meant for all low-level build • Host was added to represent • Packages required to run the package system libraries needed for the packages “that need to be compilation specific to the target platform • Installed automatically when the when the target…is not built package is installed • Anything that provides ‘sysroot’ necessarily the same as the • Stored as metadata in the binary files which is distributed via the native build platform.” Anaconda Cloud • These packages need to be able • Fermi lists the vast majority of • Good practice is to pin required to run on the build machine but its dependencies here versioning information to the be capable of outputting builds package. for a target platform • ’Host’ and ‘Build’ prefixes are • As of Conda-Build 3 you can almost always separate except augment packages in the build and in a few specific cases host sections with the ’run_exports’ header to negate the need for this section. (Weak vs. Strong) Conda Build Environments Build Host Run • Meant for all low-level build • Host was added to represent • Packages required to run the package system libraries needed for the packages “that need to be compilation specific to the target platform • Installed automatically when the when the target…is not built package is installed • Anything that provides ‘sysroot’ necessarily the same as the • Stored as metadata in the binary files which is distributed via the native build platform.” Anaconda Cloud • These packages need to be able • Fermi lists the vast majority of • Good practice is to pin required to run on the build machine but its dependencies here versioning information to the be capable of outputting builds package. for a target platform • ’Host’ and ‘Build’ prefixes are • As of Conda-Build 3 you can almost always separate except augment packages in the build and in a few specific cases host sections with the ’run_exports’ header to negate the need for this section. (Weak vs. Strong) Build Machine Conda Build Environments Build Host Run • Meant for all low-level build • Host was added to represent • Packages required to run the package system libraries needed for the packages “that need to be compilation specific to the target platform • Installed automatically when the when the target…is not built package is installed • Anything that provides ‘sysroot’ necessarily the same as the • Stored as metadata in the binary files which is distributed via the native build platform.” Anaconda Cloud • These packages need to be able • Fermi lists the vast majority of • Good practice is to pin required to run on the build machine but its dependencies here versioning information to the be capable of outputting builds package. for a target platform • ’Host’ and ‘Build’ prefixes are • As of Conda-Build 3 you can almost always separate except augment packages in the build and in a few specific cases host sections with the ’run_exports’ header to negate the need for this section. (Weak vs. Strong) Build Machine Target Machine Target Machine Conda Macros Conda build helpfully provides a number of Jinja2 functions which help automate and generalize the build process. Common Jinja2 use cases • Automatic compiler selection/setup • Pinning expressions • Templating Conda Macros Custom Variable Setting Conda build helpfully provides a number of Jinja2 functions which help Variable Calling automate and generalize the build process. Common Jinja2 use cases • Automatic compiler selection/setup • Pinning expressions • Templating Conda Macros Custom Variable Setting Conda build helpfully provides a number of Jinja2 functions which help Variable Calling automate and generalize the build process. Shell Common Jinja2 use cases Environment • Automatic compiler selection/setup reference • Pinning expressions • Templating Conda Macros Custom Variable Setting Conda build helpfully provides a number of Jinja2 functions which help Variable Calling automate and generalize the build process. Shell Common Jinja2 use cases Environment • Automatic compiler selection/setup reference • Pinning expressions • Templating Conda Compiler function Conda Compilers As of Anaconda 5 Conda provides a set of compilers which are Linux macOS recommended for use in build recipes. gcc_liunx_64 clang_liunx_64 g++_linux_64 clangxx_linux_64 Use of the Anaconda compilers gfortran_linux_64 gfortran_linux_64 helps make the recipe more agnostic with regards to the host machine. • Explicitly built for cross- compilation • Customizable Preprocessing Selectors Preprocessing selectors can be used to specify different meta-data cases for different build environments. This is most commonly used to specify different platforms/architectures as build/dependency targets. Almost all of the selector variables are Booleans and allow for logical statements to be passed in preprocessing. However, good practice is to employ comparison operators over specific selector variables. Selectors always follow a target in the format ‘TARGET # [ SELECTOR ]’ Build.sh • Default: Bash Script (.bat on Build Environment Variables Windows) • Specialized environmental • Executed in a special “Conda variables defined in the Build” environment Conda Build process. • Custom/explicit • Some are inherited and some environmental variables need are defined by Conda to be defined via meta.yaml PREFIX – Path to the build • Needed compiler flags are directory. Used by all systems specified here • MacOS/Windows have unique • Can use any installed variables scriptable build system (make, sCons, etc.) Distribution Anaconda Cloud is the primary distribution mechanism for the Conda Package manager. Organizations can have dedicated channels to distribute software built and packaged using Conda Build. Hosting for public projects are free. Private plans are available for a fee. Fermi has its own organization (the Fermi Channel) which distributes software which is developed and maintained directly by the Fermi Science Support Center Conda-Forge is another such organization. Anaconda Channels Channels organize packages by the user or group of users (Organization) that uploaded them. Label help differentiate different different packages hosted in a channel, effectively creating ‘sub-channels’. Label checking in Conda is strict. Including a label tag in a conda install or update command will search for matching packages with that label alone. Conda Forge Conda Forge Standardized Build Package Community Driven Process Centralization • Est 2016 • Conda Forge uses a Github • All feedstocks hosted on github integrated CI system in the conda-forge organization • Conda Forge seeks to expand upon the default packages • Linux -> Circle CI/Azure • Compiled packages hosted on uploaded by Anaconda Inc. the Conda-Forge Anaconda • MacOS -> Travis CI/Azure cloud channel • Packages are maintained by their uploaders in accordance • 6500+ packages currently with Conda Forge build availible standards