Cmake Refactoring

Total Page:16

File Type:pdf, Size:1020Kb

Cmake Refactoring CMake refactoring P. Hristov 19/03/2014 History I Recursive makefiles (F.Carminati): 1999-2001 Problems in dependencies Slow "Recursive Makefiles Considered Harmful" => flat makefiles similar to what Root team developed (B.Polishchuk, J.E.Revsbeck et al.): 2001-2010 Problem in dependencies if one removes files from the project make –j… doesn’t work Very often one has to clean the full project CMake: considered for the first time after ALICE-FAIR Computing Meeting 04/2008 (see the backup slides) 2 History II CMake: first implementation in 2009, converted “flat makefiles” (F.Carminati, F.Uhlig) complex processing macros, reusing lists from the “flat makefiles” missing targets CMake: second implementation in 2010 (J.Jose) simplification additional targets first implementation of CTest/CDash (not completed) CMake: production version (A.Goel, S.Tangudu) 2011-2012 all targets reimplementation of CTest/CDash (not completed) more simplifications, improved macros 3 CMake: current status After 3 years in production it is time to evaluate the situation The parallel build is working in most of the cases Still some issues with the dependencies In case one uses GLOB For header files that are copied in include subdirectory In case of changes in the installation place We are not able to profit from the automatic project generation Often recompilation triggered by the current versioning 4 CMake: current status Several targets are error prone and often need manual intervention, i.e. the RPM generation for DA and AliMDC, PAR files Custom SPEC files created via “echo” The dependencies are not tracked, the compilation always starts from the beginning The version of the RPM has to be improved The static build is not fully operational: the library is prepared, but the static AliRoot executables are not OK The code checker and the reverse engineering tool are not operational (problem with some software the depend on) 5 CMake: ideas for refactoring Configuration Avoid the use of obscure environment variables (ROOTSYS, ALICE_ROOT) as “hardcoded places” Improve the FindXXX functions. We have also many examples outside of AliRoot: FairRoot, ROOT, Geant4, AMORE, etc. Use as much as possible templates (something.in) for creation of files (SPEC, PAR, installation scripts) Re-implement the AliRoot versioning and introduce requirements on the versions of the external packages (ROOT, Geant3, Geant4, BOOST, CGAL, FASTJET) Introduce feature-based compilation, i.e. if we do not have FORTRAN compiler, switch off the compilation of some modules. 6 CMake: ideas for refactoring Structure: use “native” CMake style (see the CMake tutorials): One sub-directory per library/binary with the same name (i.e. libSTEER STEER, aliroot aliroot), or with the original name if it exists No GLOB…, use recursive add_subdirectory with explicit list of subdirectories better tracking of dependencies Explicit list of source files Possibility to use library-specific compilation options Explicit inter-library dependencies in the list of include directories Possibility to build in any location Creation of distributions with CPack: RPM, deb, … Possibility to generate projects: Xcode, MS Visual Studio, Kdevelop, etc. Possibility to generate rootmap files for automatic loading of libraries Use of CTest/CDash 7 CMake refactoring: prototyping In preparation for ROOT6 a prototype for the updated CMake build system was created in r6dev branch Includes most of the modules and libraries Some of the ideas on the structure are already implemented The copy/paste approach has to be replaced by functions/macros The build is significantly faster Several targets have to be (re-)implemented PAR files Detector algorithms Static build Rule checker Documentation CTest/CDash 8 Plan Start after the v5-05-Release Move the code of each library/binary in the corresponding subdirectory. Modify the “old” CMake build system at this stage Check other packages with CMake build for ideas and code Improve the FindXXX functions: can be done in parallel Implement the new functions, macros and templates: can be done in parallel Re-implement existing targets Branch from the master and prepare the replacement of *.pkg by CMakeLists.txt for each library/binary Estimated time: one month with two people Your help is very welcome! 9 Backup 10 CMake - What is it? • Open source project (BSD style license) • Family of tools to build, test and package software • Meta build tool generates input for native tools – UNIX Makefile – Xcode – Visual Studio 6,7,8,9 IDE files – KDevelop – Eclipse • Who is using it? – KDE, Scribus, SecondLife, ITK,VTK, FairRoot ;-) • Who is behind CMake – Kitware, Los Alamos National Labs, Sandia National Labs, National Library of Medcine, NAMIC 28.04.2008 FAIR-Alice Workshop F.Uhlig 11 CMake Features • Support complex custom commands – Generate code during build process which is then compiled (e.g. rootcint) – RuleChecker – Doxygen • Optional component support (turn on/off features) • Shared library and DLL support (version support) • Single and simple input format for all platforms • Automatic dependency generation (C, C++, Fortran) – Full dependencies: build a target in one directory, and everything this target depends on will be up to date • Parallel builds (if supported by the native tool e.g. gmake -j4) • Out of Source builds • Linux, Mac OS X, SunOS, HPUX, IRIX, Windows, etc. • Simple marco language • Only depends on compiler and native build tool 28.04.2008 FAIR-Alice Workshop F.Uhlig 12 CMake Features (cont.) • Color and progress output for make • Automatic rerun of cmake if any cmake input file changes • Graphviz output for visualization of dependency trees • Works with parallel make and on build farms • make help shows all possible targets in the directory • make foo.o build only foo.o and everything foo.o depends on • CMake has a GUI layer for easy editing of input variables • CMake has a command line interface • Cross compiling support (CMake 2.6) 28.04.2008 FAIR-Alice Workshop F.Uhlig 13 .
Recommended publications
  • C/C++ Programming with Qt 5.12.6 and Opencv 4.2.0
    C/C++ programming with Qt 5.12.6 and OpenCV 4.2.0 Preparation of the computer • Download http://download.qt.io/archive/qt/5.12/5.12.6/qt-opensource-windows- x86-5.12.6.exe and http://www.ensta-bretagne.fr/lebars/Share/OpenCV4.2.0.zip (contains OpenCV with extra modules built for Visual Studio 2015, 2017, 2019, MinGW Qt 5.12.6 x86, MinGW 8 x64), run Qt installer and select Qt\Qt 5.12.6\MinGW 7.3.0 32 bit and Qt\Tools\MinGW 7.3.0 32 bit options and extract OpenCV4.2.0.zip in C:\ (check that the extraction did not create an additional parent folder (we need to get only C:\OpenCV4.2.0\ instead of C:\OpenCV4.2.0\OpenCV4.2.0\), right-click and choose Run as administrator if needed). For Linux or macOS, additional/different steps might be necessary depending on the specific versions (and the provided .pro might need to be tweaked), see https://www.ensta-bretagne.fr/lebars/Share/setup_opencv_Ubuntu.pdf ; corresponding OpenCV sources : https://github.com/opencv/opencv/archive/4.2.0.zip and https://github.com/opencv/opencv_contrib/archive/4.2.0.zip ; Qt Linux 64 bit : https://download.qt.io/archive/qt/5.12/5.12.6/qt-opensource-linux-x64-5.12.6.run (for Ubuntu you can try sudo apt install qtcreator qt5-default build-essential but the version will probably not be the same); Qt macOS : https://download.qt.io/archive/qt/5.12/5.12.6/qt-opensource-mac-x64-5.12.6.dmg .
    [Show full text]
  • Eclipse Webinar
    Scientific Software Development with Eclipse A Best Practices for HPC Developers Webinar Gregory R. Watson ORNL is managed by UT-Battelle for the US Department of Energy Contents • Downloading and Installing Eclipse • C/C++ Development Features • Fortran Development Features • Real-life Development Scenarios – Local development – Using Git for remote development – Using synchronized projects for remote development • Features for Utilizing HPC Facilities 2 Best Practices for HPC Software Developers webinar series What is Eclipse? • An integrated development environment (IDE) • A platform for developing tools and applications • An ecosystem for collaborative software development 3 Best Practices for HPC Software Developers webinar series Getting Started 4 Best Practices for HPC Software Developers webinar series Downloading and Installing Eclipse • Eclipse comes in a variety of packages – Any package can be used as a starting point – May require additional components installed • Packages that are best for scientific computing: – Eclipse for Parallel Application Developers – Eclipse IDE for C/C++ Developers • Main download site – https://www.eclipse.org/downloads 5 Best Practices for HPC Software Developers webinar series Eclipse IDE for C/C++ Developers • C/C++ development tools • Git Integration • Linux tools – Libhover – Gcov – RPM – Valgrind • Tracecompass 6 Best Practices for HPC Software Developers webinar series Eclipse for Parallel Application Developers • Eclipse IDE for C/C++ Developers, plus: – Synchronized projects – Fortran development
    [Show full text]
  • The WAF Build System
    The WAF build system The WAF build system Sebastian Jeltsch Electronic Vision(s) Kirchhoff Institute for Physics Ruprecht-Karls-Universität Heidelberg 31. August 2010 Sebastian Jeltsch The WAF build system 31. August 2010 1 / 19 The WAF build system Introduction WorkBuildflow Sebastian Jeltsch The WAF build system 31. August 2010 2 / 19 make = major pain What we expect from our build system: flexibility integration of existing workflows access to well established libraries extensibility power usability The WAF build system Introduction WorkBuildflow For us: low-level code many many layers Sebastian Jeltsch The WAF build system 31. August 2010 3 / 19 What we expect from our build system: flexibility integration of existing workflows access to well established libraries extensibility power usability The WAF build system Introduction WorkBuildflow For us: low-level code many many layers make = major pain Sebastian Jeltsch The WAF build system 31. August 2010 3 / 19 The WAF build system Introduction WorkBuildflow For us: low-level code many many layers make = major pain What we expect from our build system: flexibility integration of existing workflows access to well established libraries extensibility power usability Sebastian Jeltsch The WAF build system 31. August 2010 3 / 19 The WAF build system Introduction Autotools (GNU Build System) GNU Build System + few dependencies on user side (shell scripts) developer autoscan ed + generates standard make files + widely used configure.ac Makefile.am – platform dependent (bash aclocal autoheader automake scripts) aclocal.m4 config.h.in Makefile.in – autoconf-configure is slow autoconf Often: tconfigure >> tmake. – another scripting language configure Makefile make user Sebastian Jeltsch The WAF build system 31.
    [Show full text]
  • Kdesrc-Build Script Manual
    kdesrc-build Script Manual Michael Pyne Carlos Woelz kdesrc-build Script Manual 2 Contents 1 Introduction 8 1.1 A brief introduction to kdesrc-build . .8 1.1.1 What is kdesrc-build? . .8 1.1.2 kdesrc-build operation ‘in a nutshell’ . .8 1.2 Documentation Overview . .9 2 Getting Started 10 2.1 Preparing the System to Build KDE . 10 2.1.1 Setup a new user account . 10 2.1.2 Ensure your system is ready to build KDE software . 10 2.1.3 Setup kdesrc-build . 12 2.1.3.1 Install kdesrc-build . 12 2.1.3.2 Prepare the configuration file . 12 2.1.3.2.1 Manual setup of configuration file . 12 2.2 Setting the Configuration Data . 13 2.3 Using the kdesrc-build script . 14 2.3.1 Loading project metadata . 14 2.3.2 Previewing what will happen when kdesrc-build runs . 14 2.3.3 Resolving build failures . 15 2.4 Building specific modules . 16 2.5 Setting the Environment to Run Your KDEPlasma Desktop . 17 2.5.1 Automatically installing a login driver . 18 2.5.1.1 Adding xsession support for distributions . 18 2.5.1.2 Manually adding support for xsession . 18 2.5.2 Setting up the environment manually . 19 2.6 Module Organization and selection . 19 2.6.1 KDE Software Organization . 19 2.6.2 Selecting modules to build . 19 2.6.3 Module Sets . 20 2.6.3.1 The basic module set concept . 20 2.6.3.2 Special Support for KDE module sets .
    [Show full text]
  • 1. Install Homebrew 2. Install Cmake 3. Build and Run the Opengl Program
    NYU Tandon School of Engineering CS6533/CS4533 Zebin Xu [email protected] Compiling OpenGL Programs on macOS or Linux using CMake This tutorial explains how to compile OpenGL programs on macOS using CMake – a cross-platform tool for managing the build process of software using a compiler- independent method. On macOS, OpenGL and GLUT are preinstalled; GLEW is not needed as we will use the core profile of OpenGL 3.2 later when we use shaders; Xcode is not required unless you prefer programming in an IDE. At the end we also discuss how to compile on Linux. Contents/Steps: 1. Install Homebrew 2. Install CMake via Homebrew 3. Build and run the OpenGL program 3.1 Build via the command line by generating Unix Makefiles (without Xcode) 3.2 Build via the Xcode IDE by generating an Xcode project (so that you can write your code in Xcode if you have it installed) 4. Compilation on Linux 5. Notes 1. Install Homebrew Homebrew is a pacKage manager for macOS. If you have installed Homebrew before, sKip this step. To install Homebrew, simply paste the command from https://brew.sh into your terminal and run. Once you have installed Homebrew, type “brew” in your terminal to checK if it’s installed. We will use Homebrew to install CMake. 2. Install CMaKe I strongly suggest installing CMake via Homebrew as it will also picK up any related missing pacKages during installation (such as installing a needed command line tool for Xcode even if you don’t have Xcode). If you have installed CMake, just sKip this step.
    [Show full text]
  • Cmake / Ctest / Cpack
    Building, Testing, and Deploying Software in a Cross-Platform Development Environment Julien Jomier [email protected] About Kitware • Software Development Company • Founded in 1998 • 110+ employees and growing • Focus in – Scientific computing – Large data visualization – Medical image processing – Informatics – Computer vision – Scientific data publishing – Quality software processes • Primarily work in Open Source About Kitware • Provide training and support • Do custom development • Sell technical books • Host and maintain projects including: VTK The Visualization Toolkit CMake A platform agnostic build system ITK Insight Segmentation and Registration Toolkit ParaView A parallel visualization application • Primary (but not sole) developers of each project Kitware: Core Technologies Overview • What is CMake? • Building with CMake • Testing with CTest/CDash • Packaging with CPack What is CMake? • CMake is the cross-platform, open-source build system that lets you use the native development tools you love the most. • It’s a build system generator • It takes plain text files as input that describe your project and produces project files or make files for use with a wide variety of native development tools. • Family of Software Development Tools – Build – CMake – Test – CTest/CDash – Package – CPack CMake: History • Built for the Insight Segmentation and Registration Toolkit (ITK) http://www.itk.org • Funded by National Library of Medicine (NLM): part of the Visible Human Project • Release-1-0 branch created in late 2001 • Other
    [Show full text]
  • Cmake Build System Distribute Your Software Easily
    CMake build system Distribute your software easily Cédric Castagnède Mars 2016 engineer innovate integrate Outline 1. Motivations of a build system 2. CMake build system 3. Test integration 4. Packaging an application 5. Automation of all steps 6. Conclusions and opened discussion engineer innovate integrate Motivations of a build system engineer innovate integrate What problems do build system solve? For a developer: • reduce the time spent in the cycle “edit / compile / test” cycle • compile only what is necessary in the source code For a development team: • generate packages • run tests • generate documentation For a user: • install software easily • have understandable error during install phase • tune installation engineer innovate integrate Build a software: a lot of evil ways Examples: • “I will do a script to launch all my command and it will be ok” • system-dependent, all path dependent, etc. • high cost for developers and users • “I will do a makefile with a make.inc, my software earns portability” • costly for the user: manual configuration • portable ≠ customizable • “I will do a makefile with several make.inc, my software covers all software stacks” • costly for everyone: any make.inc won’t fit the situation • examples ≠ software stack supports • software stack supports = test expected behavior • Etc. engineer innovate integrate Features of a build system (1) automatic dependency management of source code • compile only the modified sources files and tiers dependencies software portability • use native build environment • determine available OS/compiler features : foo.h, libbar, strndup, -Wall, etc. • name correctly the library: .so / .dylib / .dll adaptability according user environment • auto-configuration of the project • determine the availability and location of libraries, commands, etc… engineer innovate integrate Features of a build system (2) customize installation • cross-compiling • give some information: --help • possibility to set information: --prefix, --libdir, --disable-shared, etc.
    [Show full text]
  • Kdevelop Handbook
    KDevelop Handbook This documentation was converted from the KDE UserBase KDevelop4/Manual page. KDevelop Handbook 2 Contents 1 What is KDevelop?6 2 Sessions and projects: The basics of KDevelop8 2.1 Terminology . .8 2.2 Setting up a session and importing an existing project . .9 2.2.1 Option 1: Importing a project from a version control system server . .9 2.2.2 Option 2: Importing a project that is already on your hard drive . 10 2.3 Setting up an application as a second project . 10 2.4 Creating projects from scratch . 10 3 Working with source code 12 3.1 Tools and views . 12 3.2 Exploring source code . 14 3.2.1 Local information . 14 3.2.2 File scope information . 16 3.2.3 Project and session scope information . 17 3.2.4 Rainbow color highlighting explained . 19 3.3 Navigating in source code . 19 3.3.1 Local navigation . 19 3.3.2 File scope navigation and outline mode . 20 3.3.3 Project and session scope navigation: Semantic navigation . 21 3.4 Writing source code . 25 3.4.1 Auto-completion . 25 3.4.2 Adding new classes and implementing member functions . 27 3.4.3 Documenting declarations . 31 3.4.4 Renaming variables, functions and classes . 34 3.4.5 Code snippets . 35 3.5 Modes and working sets . 37 3.6 Some useful keyboard shortcuts . 39 KDevelop Handbook 4 Code generation with templates 41 4.1 Creating a new class . 41 4.2 Creating a new unit test . 43 4.3 Other files .
    [Show full text]
  • Worch Build System for LBNE Software
    Worch Build System for LBNE Software Brett Viren Physics Department LBNE Pre-meeting Workshop 2014/1/31 Brett Viren (BNL) worch/lbne 2014/1/31 1 / 16 Outline Overview Basic Use Some details LBNE-Specific To Do Brett Viren (BNL) worch/lbne 2014/1/31 2 / 16 Overview What is worch worch is a system for: build orchestration worch is a meta-build system for the installation of a suite of software by scheduling the execution of the installation commands that are native to each package (configure, make, etc). configuration management all configuration information for the suite is captured in a few text files allowing for release management, reproducible installation. All installation policy/convention is expressed in the configuration. reliable build automation once started, runs fully unattended unless an error occurs in which case the failure is made clear with log files for postmortem and a generated shell to reproduce the failure in-situ. Brett Viren (BNL) worch/lbne 2014/1/31 3 / 16 Overview The pieces of worch Worch is made up of these parts: waf a build tool (“make on steroids”) and an engine for dependency resolution and parallel execution. Implemented in pure-Python. A copy is included with worch. orch a Python module extending waf and providing worch’s functionality. Additional, suite-specific modules may also be needed. configuration simple text files that describe a software suite to install. When performing an installation the user needs to bring these three parts together. Brett Viren (BNL) worch/lbne 2014/1/31 4 / 16 Basic Use A Simple Example - cmake + GNU bc & hello worch provides several example software suites configurations $ git clone https://github.com/brettviren/worch.git $ cd worch/ $ ./waf --prefix=/tmp/worch-simple-example \ (1) --out=tmp (2) --orch-config=examples/simple/*.cfg \ (3) configure (4) $ ./waf (5) (1) Sets the root directory for the final installation.
    [Show full text]
  • Cmake - Cross-Platform Make
    CMake - Cross-Platform Make R. Douglas Barbieri Made to Order Software Corporation Introduction What is CMake? Control the software compilation process using simple platform-independent and compiler- independent configuration files Generate native makefiles and workspaces that can be used in the compiler environment of your choice Provides packaging (using CPack) and testing (using CTest) Who is using it? Linden Lab (for their Second Life project) KDE4 Boost MySQL The Half-Life 2 SDK Rosegarden Loads of others (see http://www.cmake.org/Wiki/CMake_Projects) for more details Why CMake? Better than other build systems: Custom GNU/Makefiles autoconf/automake CodeBlocks other FOSS IDEs (sadly, not DevC++) Microsoft Visual Studio and other proprietary IDEs Apple XCode Why CMake (con't) Can target multiple compilers, build systems and IDEs using a single set of configuration files Other build systems are difficult to set up and debug (particulary autoconf/automake) Has a simple to use language to allow customization for multiple platforms with relative ease. Great even for a single platform! How Does it Work? CMake uses ”Generators” to create your target build files Uses configuration files to target your particular system Uses your custom config or existing rules to locate and build against third party libraries Provides a simple language to help customize for platform-specific idioms Generators What are CMake Generators? They can produce make/project files for many different IDEs, GNU/Make and Microsoft's Nmake Customized for your specific platform Able to produce tailored project files specific to your favorite development IDE or system Lots are available! Generators GNU/Linux: $ cmake The following generators are available on this platform: Unix Makefiles Generates standard UNIX makefiles.
    [Show full text]
  • Package Management for Android C++ by Alex Cohn
    Package Management for Android C++ by Alex Cohn Why Prefab? Android++ (reminder) ● Android OS is a Linux distribution according to the Linux Foundation ● Toolchain: clang version 8.0.7 (9.0.8 in next beta) ● Standard library: LLVM c++ ● Build system: Gradle https://developer.android.com/studio/projects/add-native-code ● Java Native Interface libraries used in Java apps ● JNI binaries distribution: Maven AAR dependencies https://developer.android.com/studio/projects/android-library Challenge++ for Android developers ● Most developers only work in Kotlin or Java, but this may be not enough ● Most developers never leave Android Studio, but this may be not enough ● It’s easy to use a prebuilt JNI library, but this may be not enough ● It’s easy to add your C++ code to Android project, but this may be not enough The challenge: third-party C++ dependencies! Challenge++ for library authors ● Need to maintain two build systems: ndk-build and CMake ● Need to support different build platforms, Windows being more painful ● Need to take care of library dependencies ○ Either bundle all dependencies in a huge source tree (and be locked out of updates) ○ Or link to dependencies and let the app developer suffer We need an integrated and flexible package management system. Prefab comes to rescue (1/3) https://android-developers.googleblog.com/2020/02/native-dependencies-in-android-studio-40.html root/build.gradle buildscript { dependencies { classpath 'com.android.tools.build:gradle:4.0.0-beta04' } } app/build.gradle dependencies { implementation 'com.arthenica:mobile-ffmpeg-prefab-min:4.3.1' } Prefab comes to rescue (2/3) CMakeLists.txt find_package(ffmpeg REQUIRED CONFIG) add_library(app SHARED app.cpp) target_link_libraries(app ffmpeg::libavcodec) Android.mk LOCAL_SHARED_LIBRARIES += libavcodec # libavutil will be available, too $(call import-module,prefab/ffmpeg) Prefab comes to rescue (3/3) app.cpp #include "libavcodec/avcodec.h" This prefab package used make install to prepare the public headers for ffmpeg.
    [Show full text]
  • Notepad++ User Manual
    npp-user-manual Table of Contents 1. Introduction 2. File Management 3. Session Management 4. Editing 5. Document Properties 6. Searching 7. Display 8. Macros 9. Commands 10. Windows Dialog 11. GUI Elements 12. Switching between Documents 13. Languages 14. Settings 15. Plugins 16. Command Line 17. Control Files 18. Shell Extension 19. Upgrading 20. Further Help 2 npp-user-manual Notepad++ User Manual Notepad++ User Manual This document is dedicated to the countless lost hours of both people having to search for simple answers, and people having to answer questions that shouldn't have to be asked in the first place. Documentation is always hard to get, good documentation nearly impossible. This markup is based in the default appearance of Microsoft Windows. This can be different on your own system, but the names (text) should always the same. This is especially important on the screenshots. They are only for reference, your appearance of Notepad++ can be quite different. All the images used in this helpfile assume the default settings. If you change any of these settings, you'll have to look for the other image instead, this is most noticeable with the toolbar (the position of images will always remain the same). Also, when talking about the left and right mouse buttons, the logical left and right is meant. These are the same as the physical left and right mouse buttons in the case of a right-handed setup. However, if you have a left-handed setup you will probably have the buttons swapped so act accordingly. This means in general you do what you normally do for most actions, Notepad++ mostly follows the standard Windows application behavior.
    [Show full text]