Intel® Xeon Phi™ Coprocessor System Software Developers Guide SKU# 328207-003EN March, 2014 Intel® Xeon Phi™ System Software Developer’s Guide INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked “reserved” or “undefined.” Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The Intel® MIC Architecture coprocessors described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, Intel literature may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm Intel, the Intel logo, Intel® Pentium®, Intel® Pentium® Pro, Xeon®, Intel® Xeon Phi™, Intel® Pentium® 4 Processor, Intel Core™ Solo, Intel® Core™ Duo, Intel Core™ 2 Duo, Intel Atom™, MMX™, Intel® Streaming SIMD Extensions (Intel® SSE), Intel® Advanced Vector Extensions (Intel® AVX), Intel® VTune™ Amplifier XE are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright 2011-2014 Intel Corporation. All rights reserved. Page 2 Intel® Xeon Phi™ System Software Developer’s Guide Table of Contents Table of Contents ......................................................................................................................... 3 List of Figures ............................................................................................................................... 6 List of Tables ................................................................................................................................ 7 1 Introduction .................................................................................................................... 9 1.1 Programming Model ..................................................................................................... 9 1.1.1 Application Programming .......................................................................... 9 1.1.2 System Programming ................................................................................ 9 1.2 Section Overview ........................................................................................................ 10 1.3 Related Technologies and Documents ....................................................................... 10 2 Intel® Xeon Phi™ Coprocessor Architecture .................................................................... 12 2.1 Intel® Xeon Phi™ Coprocessor Architecture ............................................................... 12 2.1.1 Core ......................................................................................................... 15 2.1.2 Instruction Decoder ................................................................................. 17 2.1.3 Cache Organization and Hierarchy .......................................................... 18 2.1.4 Page Tables .............................................................................................. 21 2.1.5 Hardware Threads and Multithreading ................................................... 22 2.1.6 Faults and Breakpoints ............................................................................ 23 2.1.7 Performance Monitoring Unit and Events Monitor ................................ 24 2.1.8 System Interface ...................................................................................... 26 2.1.9 VPU and Vector Architecture .................................................................. 32 2.1.10 Intel® Xeon Phi™ Coprocessor Instructions ............................................ 33 2.1.11 Multi-Card ............................................................................................... 33 2.1.12 Host and Intel® MIC Architecture Physical Memory Map ....................... 34 2.1.13 Power Management ................................................................................ 35 2.2 Intel® Xeon Phi™ Coprocessor Software Architecture................................................ 36 2.2.1 Architectural Overview............................................................................ 36 2.2.2 Intel® Manycore Platform Software Stack (MPSS) .................................. 39 2.2.3 Bootstrap ................................................................................................. 40 2.2.4 Linux* Loader .......................................................................................... 41 Page 3 Intel® Xeon Phi™ System Software Developer’s Guide 2.2.5 The Coprocessor Operating System (coprocessor OS) ............................ 42 2.2.6 Symmetric Communication Interface (SCIF) ........................................... 44 2.2.7 Host Driver............................................................................................... 44 2.2.8 Sysfs Nodes .............................................................................................. 51 2.2.9 Intel® Xeon Phi™ Coprocessor Software Stack for MPI Applications ...... 53 2.2.10 Application Programming Interfaces....................................................... 65 3 Power Management, Virtualization, RAS ........................................................................ 66 3.1 Power Management (PM) ........................................................................................... 66 3.1.1 Coprocessor OS Role in Power Management ......................................... 67 3.1.2 Bootloader Role in Power Management ................................................. 68 3.1.3 Host Driver Role in Power Management ................................................. 68 3.1.4 Power Reduction ..................................................................................... 69 3.1.5 PM Software Event Handling Function ................................................... 87 3.1.6 Power Management in the Intel® MPSS Host Driver .............................. 89 3.2 Virtualization ............................................................................................................... 93 3.2.1 Hardware Assisted DMA Remapping ...................................................... 93 3.2.2 Hardware Assisted Interrupt Remapping ................................................ 93 3.2.3 Shared Device Virtualization ................................................................... 93 3.3 Reliability Availability Serviceability (RAS) .................................................................. 93 3.3.1 Check Pointing ......................................................................................... 94 3.3.2 Berkeley Labs Check point and Restore (BLCR) ....................................... 95 3.3.3 Machine Check Architecture (MCA) ........................................................ 97 3.3.4 Cache Line Disable ................................................................................. 112 3.3.5 Core Disable .......................................................................................... 112 3.3.6 Machine Check Flows ............................................................................ 113 3.3.7 Machine Check Handler ........................................................................ 114 3.3.8 Error Injection........................................................................................ 115 4 Operating System Support and Driver Writer’s Guide.................................................... 117 4.1 Third Party OS Support ............................................................................................. 117 4.2 Intel®