Facebook Server Intel Motherboard V4.0 Project Tioga Pass Rev 0.30 Author: Whitney Zhao, Hardware Engineer, Facebook Jia Ning, Hardware Engineer, Facebook 1 Revision History Table 1-1 Date Name Description 6/3/2015 Jia Ning - Version 0.1 release 7/29/201 Whitney - Version 0.2 release 6 Zhao - Updated Figure and Tables References - Update contents 1/31/201 Whitney - Version 0.3 release for OCP summit 7 Zhao - Update contents - Minor corrections 2 January 2017 Open Compute Project Tioga Pass v4.0 © 2016 Facebook. As of July 26, 2016, the following persons or entities have made this Specification available under the Open Compute Project Hardware License (Permissive) Version 1.0 (OCPHL-P), which is available at http://www.opencompute.org/.../spec-submission-process/. Facebook, Inc. Your use of this Specification may be subject to other third party rights. THIS SPECIFICATION IS PROVIDED "AS IS." The contributors expressly disclaim any warranties (express, implied, or otherwise), including implied warranties of merchantability, non-infringement, fitness for a particular purpose, or title, related to the Specification. The Specification implementer and user assume the entire risk as to implementing or otherwise using the Specification. IN NO EVENT WILL ANY PARTY BE LIABLE TO ANY OTHER PARTY FOR LOST PROFITS OR ANY FORM OF INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF ANY CHARACTER FROM ANY CAUSES OF ACTION OF ANY KIND WITH RESPECT TO THIS SPECIFICATION OR ITS GOVERNING AGREEMENT, WHETHER BASED ON BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE), OR OTHERWISE, AND WHETHER OR NOT THE OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE." http://opencompute.org 3 2 Scope This specification describes Facebook dual sockets server Intel Motherboard v4.0 (Project name: Tioga Pass) design and design requirement to integrate Intel Motherboard v4.0 into Open Rack V21. 3 Contents 1 Revision History ......................................................................................................................... 2 2 Scope ......................................................................................................................................... 4 3 Contents .................................................................................................................................... 4 4 Overview.................................................................................................................................... 8 4.1 Overview ...................................................................................................................... 8 4.2 Open Rack Introduction ............................................................................................... 8 5 Physical Specifications ............................................................................................................... 9 5.1 Block Diagram ............................................................................................................... 9 5.2 Placement and Form Factor ......................................................................................... 9 5.3 CPU and Memory ....................................................................................................... 10 5.4 Platform Controller Hub (PCH) ................................................................................... 11 5.5 PCIe Usage .................................................................................................................. 12 5.6 PCB Stack‐Up .............................................................................................................. 12 6 BIOS ......................................................................................................................................... 16 6.1 BIOS Chip .................................................................................................................... 16 6.2 BIOS Source Code ....................................................................................................... 16 6.3 BIOS Feature Requirements ....................................................................................... 16 7 PCH Intel® SPS Firmware Plan of Record ................................................................................. 22 8 BMC ......................................................................................................................................... 23 8.1 Management Network Interface ................................................................................ 23 8.2 Local Serial Console and SOL ...................................................................................... 23 8.3 Graphic and GUI ......................................................................................................... 24 8.4 Remote Power Control and Power policy .................................................................. 24 8.5 Port 80 POST ............................................................................................................... 24 8.6 Power and System Identification LED ........................................................................ 24 1 http://files.opencompute.org/oc/public.php?service=files&t=348f3df2cc4ce573397fcc4424f68ca6&download 4 January 2017 Open Compute Project Tioga Pass v4.0 8.7 Platform Environment Control Interface (PECI) ......................................................... 25 8.8 Power and Thermal Monitoring and power limiting .................................................. 25 8.9 SMBUS Diagram ......................................................................................................... 25 8.10 Sensors/Events ........................................................................................................... 25 8.11 SEL .............................................................................................................................. 33 8.12 FSC in BMC.................................................................................................................. 34 8.13 OEM commands ......................................................................................................... 35 8.14 BMC FW chip and Firmware Update .......................................................................... 37 8.15 BMC Update Dual PCH flash ....................................................................................... 37 8.16 BMC Update and Access CPLD ................................................................................... 39 8.17 BMC Time Sync ........................................................................................................... 39 8.18 PCIe and Mezzanine card Thermal monitoring .......................................................... 39 8.19 BMC PPIN Implementation ........................................................................................ 40 8.20 BMC Average Power Reporting .................................................................................. 40 8.21 BMC Access and Update VR Firmware ....................................................................... 40 8.22 BMC MSR Dump from OOB ........................................................................................ 40 8.23 BMC Network Status .................................................................................................. 41 8.24 BMC Secure boot ........................................................................................................ 41 9 Thermal Design Requirements ................................................................................................ 43 9.1 Data Center Environmental Conditions ..................................................................... 43 9.2 Server operational condition ...................................................................................... 43 9.3 Thermal Kit Requirements.......................................................................................... 45 10 I/O System ............................................................................................................................... 47 10.1 PCIe x32 Slot/Riser Card ............................................................................................. 47 10.2 DIMM Sockets ............................................................................................................ 54 10.3 Mezzanine Card .......................................................................................................... 54 10.4 Network ...................................................................................................................... 59 10.5 USB ............................................................................................................................. 60 10.6 SATA ........................................................................................................................... 61 10.7 M.2 ............................................................................................................................. 61 10.8 Debug Header ............................................................................................................. 62 10.9 Switches and LEDs ...................................................................................................... 66 10.10 Fan connector ............................................................................................................. 67 http://opencompute.org 5 10.11 TPM Connector and Module .....................................................................................