Port, Develop, Tune on Linux on POWER8
Total Page:16
File Type:pdf, Size:1020Kb
May 22-25, 2017 | San Francisco Port, Develop, Tune on Linux on POWER8 Why Linux on POWER? ● Enterprise Class Hardware ● 2x price/performance ● Easy to port to, Easy to use ● Optimized for performance –High-performing, highly reliable platform, capable of handling large quantities of data more efficiently – Enables high speed off-load capabilities with technologies such as CAPI and GPUs. –Flexible, fast execution of analytics algorithms ● 4X threads per core vs. x86 (up to 1536 threads per system) ● Memory: Large, fast workspace maximizes business insight ● 4X memory bandwidth vs. x86 (up to 32 TB of memory) ● Cache Ensure continuous data load for fast responses ● 6X more cache vs x86 (>19 MB cache per core) Linux is Linux everywhere! Software: It's Linux! ▶Linux on POWER is Linux, with full distribution support from the major enterprise Linux distributions: – Red Hat Enterprise Linux (RHEL) from Red Hat, – Ubuntu from Canonical, and – SUSE Linux Enterprise Server (SLES) from SUSE. ● Thousands of open source binary packages are supported on the POWER platform ● Most standard packages are available directly from the major Linux distributions. ● Thousands of community maintained ppc64le packages that run on POWER ● Open Source POWER Availability Tool (OSPAT), searches ppc64le distros and ported apps The definitive guide to porting Linux applications to POWER ▶Research ▶Plan ▶Set up ▶Build ▶Resources Overview of Research Process ▶Plan for the port ▶Get started ▶Get hardware access ▶Choose Linux distribution ▶Prepare development environment ▶Determine database requirements ▶Gather test cases and tools ▶Are you ready to get started? Research May 22-25, 2017 San Francisco Researching your app – Getting Started ▶Many packages are trivial to port to POWER ▶often require a simple recompile or make command. ▶Some require minor tweaks to #ifdef configurations ▶Some have ppc64 – which implies big endian ▶Some need additional libraries, e.g. some math libraries ▶Some simply require identifying and loading the right dependencies ▶If the port is not trivial, we need to build a plan... AccesstoHardware May 22-25, 2017 San Francisco Access to POWER8 hardware ▶If you have your own hardware already, you are set ▶But if not... –There are POWER cloud resources available throughout the world –Use the OpenPOWER Developer Resources Map, ●https://developer.ibm.com/linuxonpower/cloud-resources/ Cloud centers available world wide Highlights of current clouds Highlights of current clouds ●Cloud Hardware Recommendations ● If you are an academic or open source developer consider these cloud resources: – Oregon State University- Open Source Lab (OSL) ● In partnership with IBM, the OSL provides access to IBM POWER based servers for developing and testing open source projects. Request access – Unicamp MiniCloud ● No-charge, access to Power virtual machines (VMs) for developing, testing or migrating applications to Linux on Power. Hosted by University of Campinas, Brazil. Request access ● If you are an ISV or need access to GPUs, CAPI, or NVlink technologies consider: – Nimbix ● The Nimbix Jarvice Cloud platform is the first commercial cloud environment to feature POWER8 with NVLink. It features ready to deploy instances for deep learning and NVLink application deployment and development. ● If you don’t have hardware and don’t want to use a cloud,use an emulator – QEMU (user-mode emulation) available in SDK – IBM Power Functional Simulator (full system simulation) are provided with the SDK. In this mode, you can develop and port applications without the need for Power hardware. May 22-25, 2017 San Francisco Hardware: Simulators ▶If you want to try the port through a simulator, consider: –Instal the IBM Software Developer Kit (SDK) for Linux on Power on your Linux laptop/desktop –run your Power binaries on the same desktop through emulation technologies ●Both QEMU (user-mode emulation) and the ●IBM Power Functional Simulator (full system simulation) –are provided with the SDK. In this mode, you can develop and port applications without the need for Power hardware. LinuxDistributions May 22-25, 2017 San Francisco Linux Distribution ● Do you need the latest and greatest technology? – Ubuntu, Debian, and Fedora typically have frequent releases that integrate the latest open source projects. They are good choices for initial development in cloud, containers, cognitive, HPC, and other areas where the latest technology is absolutely necessary. ● Do you need the best support? – Red Hat Enterprise Linux (RHEL), SUSE Enterprise Linux (SLES), and Ubuntu, in partnership with IBM, provide long term distribution support and are a common target for key ISV applications that expect long term stability. ● Community supported distributions – Ubuntu (and its many variants, such as kubuntu), – CentOS, and – OpenSUSe provide community supported distributions, which provide much of the same application support as their enterprise versions but without the corresponding support costs. May 22-25, 2017 San Francisco Selecting a Distribution based on apps ● Are prerequisite and dependent packages available on the distribution of your choice? – Use the Open Source POWER Availability Tool (OSPAT) search engine to see if the correct packages and versions of packages are available on the distribution of your choice. May 22-25, 2017 San Francisco Open Source POWER Availability Tool (OSpat) May 22-25, 2017 San Francisco Ospat Search for Elasticsearch May 22-25, 2017 San Francisco Ospat Future Directions ▶Currently scans common distro contents ▶Adding pointers to all packages ported to POWER ▶Planning to add Ruby gems, Python pips, Node.js npms ▶Including HPC ported applications lists ▶Adding biobuilds.org contents ▶Working with OSU Center for Genome Research and Biocomputing (CGRB) –http://cgrb.oregonstate.edu/ –Listing their projects –Hosting build instructions at github.com/ppc64le/build-scripts ▶Working with Nimbix to build docker containers ▶Populating https://hub.docker.com/u/ppc64le/ with pre-built POWER containers ▶Tracking binaries when available ▶Many Bullets ▶Logo Colored Bullets May 22-25, 2017 San Francisco Other Distributions ● There are several other Linux distributions that support the POWER platform including: – Fedora (be sure to search for ppc64le packages) – Debian – CentOS May 22-25, 2017 San Francisco Languages May 22-25, 2017 San Francisco Application Programming Language ● What language is your application written in? – Power supports most but not all current languages (see list below) – Most compiled applications will simply require a re-build – Most interpreted languages will run without changes ● C# Clang++ ada Angular.js awk bash Clang Clojure (JVM) (Mono) (LLVM) Fixedhea GNU D (LLVM) DoT.js Erlang G++ GCC GNU Go der.js Fortran Java GNU GNU Objective Go Lang Haskell HHVM (OpenJD JQuery JRuby Objective C C++ K) Node.js/V Julia (LLVM) Lua Modula 2/3 OCaml Octave Perl Phantom.js 8S Rust Sala PHP pypy Python R Lang Ruby SpiderMonkey (LLVM) (JVM) SQL Swift (LLVM) May 22-25, 2017 San Francisco Porting a Compiled Language Compiled languages Effort required to port: Recompile and test* C, C++, and FORTRAN languages are compiled to instructions for a specific machine/platform. The C language in particular can expose more of the underlying machine architecture to the program and is thus, slightly less portable. That said, most applications written in a compiled language with no platform dependencies will only require a recompile to run on Power. *It’s estimated that less than 5% of Linux applications from any platform written in C/C++ will require source code changes. The Source Code Advisor, which is part of the IBM Software Developer Kit (SDK) for Linux on Power, can analyze your source code and show you which areas require changes. The SDK is discussed in more detail below. May 22-25, 2017 San Francisco C/C++ Applications Many of the most popular compilers are available and optimized for POWER, including the following: ● GNU Compiler Collection (GCC) ● Clang ● XL C/C++ for Linux ● XL Fortran for Linux May 22-25, 2017 San Francisco IBM Advanced Toolchain for Linux on POWER Install the Advance Toolchain for the latest open source compilers, runtime libraries, and tools enabled and optimized for POWER8. Current GCC compilers and language levels Languages: C/C++, FORTRAN, Go Includes cross compilers for SDK client Optimized POSIX runtime libraries: libc, libm, libpthread Extra libraries: Zlib, OpenSSL, Boost, Tcmalloc, Intel TBB, SPHDE Performance profiling tools: oprofile, valgrind Learn more about the Advance Toolchain for Linux on Power (including how to install it). ● https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_44 6ebc23c550/page/IBM%20Advance%20Toolchain%20for%20PowerLinux%20Documentation?section=insta llation May 22-25, 2017 San Francisco IBM Software Development Kit (SDK) for Linux on Power If you’re porting an application written in C/C++, consider using the the SDK for Linux on Power, which is a free, Eclipse-based integrated development environment (IDE) that includes powerful tools to aid developers porting to Linux on Power: ● Migration Advisor ● Build Advisor ● Source Code Advisor Each of these tools, and many others (including the IBM Advance Toolchain) are integrated into the SDK, which you can download from here: https://developer.ibm.com/linuxonpower/sdk-download/ or in source form from here: https://github.com/open-power-sdk May 22-25, 2017 San Francisco Migration Advisor The Migration Advisor scans