System Event Log Troubleshooting Guide
Total Page:16
File Type:pdf, Size:1020Kb
System Event Log (SEL) Troubleshooting Guide Summary and definition of events generated by Intel® platforms. Rev. 3.6 August 2021 Intel® Server Products and Solutions <Blank page> System Event Log (SEL) Troubleshooting Guide Document Revision History Date Revision Changes January 2015 1.0 Initial release. March 2016 2.0 Added aggregate Voltage sensor information for all platforms. Added Intel® Xeon® scalable processor family support. Added Mirroring Redundancy State Sensor for Intel® Xeon® scalable processor family. March 2017 3.0 Added ECC error event define for Intel® Xeon® processor scalable family. Added memory parity error for Intel® Xeon® processor scalable family. Applied new formatting and edited for clarity. October 2017 3.1 Added aggregate Voltage sensor information for S2600BP, S2600WF, S2600ST, S2600BT December 2017 3.2 Changed description for QPI Fatal Error and Fatal Error #2 – Next Steps Update description for Correctable and uncorrectable ECC error sensor typical characteristics. April 2019 3.3 Update the Syscfg command to dump BMC debug log Add 2nd Generation Intel® Xeon® scalable processor family support. Add POST error codes for S2600WF/S2600WFR/S2600BP/S2600BPR/S2600ST/S2600STR Add support for Intel® Server Board based on Intel® Xeon® Platinum 9200 processor family. Update System Event Sensor for BIOS/Intel® ME OOB update and BIOS configuration change from BMC EWS. Add NVMe* Temperature sensor and NVMe* Critical Warning sensor support. September 2019 3.4 Add remote debug sensor support. Add system firmware security sensor support. Add bad user PWD sensor support. Add KCS policy sensor support Add support for PS3 sensor on S9200WK. November 2019 3.5 Add support for Memory ECC Uncorrectable/correctable error SEL event on S9200WK. Update BIOS owned sensor (Table 6, Table 7) August 2021 3.6 Add new tables for decoding CPU and DIMM index in relate SEL (Table 91, Table 92) 3 System Event Log (SEL) Troubleshooting Guide Disclaimers Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software, or service activation. Learn more at Intel.com, or from the OEM or retailer. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting www.intel.com/design/literature.htm. Intel, the Intel logo, Xeon, and Xeon Phi are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © Intel Corporation 4 System Event Log (SEL) Troubleshooting Guide Table of Contents 1. Introduction ............................................................................................................................................................... 13 1.1 Purpose .............................................................................................................................................................................. 13 1.2 Industry Standard .......................................................................................................................................................... 13 1.2.1 Intelligent Platform Management Interface (IPMI) ........................................................................................... 13 1.2.2 Baseboard Management Controller (BMC) .......................................................................................................... 14 1.2.3 Intel® Intelligent Power Node Manager Version 3.0 ........................................................................................ 14 2. Basic Decoding of a SEL Record ............................................................................................................................ 16 2.1 Default Values in the SEL Records .......................................................................................................................... 16 2.2 Notes on SEL Logs and Collecting SEL Information ........................................................................................ 19 2.2.1 Example of Decoding a PCIe* Correctable Error Events ................................................................................ 19 2.2.2 Example of Decoding a Power Supply Predictive Failure Event ................................................................ 20 2.2.3 Example of Decoding an NVMe* Temperature Sensor Event ..................................................................... 20 3. Sensor Cross Reference List ................................................................................................................................... 22 3.1 BMC-Owned Sensors (GID = 0020h) ...................................................................................................................... 22 3.2 BIOS POST-Owned Sensors (GID = 0001h) ........................................................................................................ 25 3.3 BIOS SMI Handler-Owned Sensors (GID = 0033h) .......................................................................................... 25 3.4 Intel® NM/Intel® ME Firmware-Owned Sensors (GID = 002Ch or 602Ch) .............................................. 25 3.5 Microsoft* OS-Owned Events (GID = 0041h) ..................................................................................................... 26 3.6 Linux* Kernel Panic Events (GID = 0021h) ........................................................................................................... 26 4. Power Subsystems ................................................................................................................................................... 27 4.1 Threshold-Based Voltage Sensors ......................................................................................................................... 27 4.2 Aggregate Voltage Fault Sensor .............................................................................................................................. 28 4.3 Voltage Regulator Watchdog Timer Sensor ....................................................................................................... 38 4.3.1 Voltage Regulator Watchdog Timer Sensor – Next Steps ............................................................................ 39 4.4 Power Unit ......................................................................................................................................................................... 39 4.4.1 Power Unit Status Sensor ........................................................................................................................................... 39 4.4.2 Power Unit Redundancy Sensor .............................................................................................................................. 40 4.4.3 Node Auto Shutdown Sensor ................................................................................................................................... 41 4.5 Power Supply ................................................................................................................................................................... 41 4.5.1 Power Supply Status Sensors ................................................................................................................................... 41 4.5.2 Power Supply Power in Sensors .............................................................................................................................. 43 4.5.3 Power Supply Current Out % Sensors .................................................................................................................. 43 4.5.4 Power Supply Temperature Sensors ..................................................................................................................... 44 4.5.5 Power Supply Fan Tachometer Sensors .............................................................................................................. 45 5. Cooling Subsystem .................................................................................................................................................. 46 5.1 Fan Sensors ...................................................................................................................................................................... 46 5.1.1 Fan Tachometer Sensors ............................................................................................................................................ 46 5.1.2 Fan Presence and Redundancy Sensors .............................................................................................................