Table of Contents
Total Page:16
File Type:pdf, Size:1020Kb
1 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Eine phatastische Reise ins Innere der Hardware Franz Haberhauer Stefan Hinker Oracle Hardware in 3D 5 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. T5 and M5 PCIe Carrier Card . Supports standard low-profile PCIe cards Air Flow PCIe Retimer x16 Connector (x8 electrical) 6 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. PCIe Data Paths: Full System . Two root complexes per T5 processor . Each PCIe port on a T5 processor controls a single PCIe slot 7 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. T5-2 Block Diagram DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM BoB BoB BoB BoB BoB BoB BoB BoB BoB BoB BoB BoB BoB BoB BoB BoB T5-0 T5-1 CPU CPU TPM Host & CPU PCIe Debug CPU PCIe Debug Data Flash DC/DCs 0 1 Port DC/DCs 0 1 Port x8 x8 FPGA x8 x4 x8 x1 HDD0 DBG SAS/SATA x1 HDD0 IO Controller x4 x4 PCIe PCIe SP Module HDD0 get rid of all inside x8 x8 SAS/SATA smallSwitch boxes 0 Switch 1 FRUID HDD0 IO Controller Sideband Mgmt DRAM HDD0 USB 1.1 Keyboard Mouse Service SPI x8 USB 3.0 x8 USB 2.0 Storage Flash HDD0 Host Processor SATA DVD NAND USB 2.0 Hub USB USB 3.0 USB Internal USB Hub VGA VGA REAR IO Board USB2 USB3 VGA USB0 USB1 VGA Serial Enet Quad 10Gig Enet DB15 Mgmt Mgmt Slot 2 (8) 2 Slot (8) 3 Slot (8) 4 Slot (8) 5 Slot (8) 6 Slot (8) 7 Slot (8) 8 Slot Slot 1 (8) 1 Slot 10/100 FAN BOARD REAR IO 8 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Agenda . Preparing a LDom Guest with it’s own Root Complex . Removing a PCIe card . Re-inserting a PCIe card 9 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Creating a Root Domain An Example on M6-32 – Listing IO Devices root@primary # ldm ls-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ pci_32 BUS pci_32 primary IOV pci_33 BUS pci_33 primary IOV pci_34 BUS pci_34 primary IOV pci_35 BUS pci_35 primary IOV pci_36 BUS pci_36 primary IOV […] /SYS/IOU2/PCIE3 PCIE pci_32 primary EMP /SYS/IOU2/EMS1/CARD/NET0 PCIE pci_32 primary OCC /SYS/IOU2/EMS1/CARD/SCSI PCIE pci_32 primary OCC /SYS/IOU2/PCIE2 PCIE pci_33 primary OCC /SYS/IOU2/PCIE5 PCIE pci_34 primary EMP /SYS/IOU2/PCIE8 PCIE pci_35 primary EMP /SYS/IOU2/PCIE11 PCIE pci_36 primary EMP /SYS/IOU2/EMS3/CARD/NET0 PCIE pci_36 primary EMP /SYS/IOU2/EMS3/CARD/SCSI PCIE pci_36 primary EMP 10 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Creating a Root Domain An Example on M6-32 – Giving away some devices root@primary # ldm start-reconf primary ldm rm-io pci_33 ldm rm-io pci_34 Reboot […] ldm add-io pci_33 mars ldm add-io pci_34 mars 11 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Creating a Root Domain Checking IO Devices again root@primary # ldm ls-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ pci_32 BUS pci_32 primary IOV pci_33 BUS pci_33 mars OCC pci_34 BUS pci_34 mars EMP pci_35 BUS pci_35 primary IOV pci_36 BUS pci_36 primary IOV […] /SYS/IOU2/PCIE3 PCIE pci_32 primary EMP /SYS/IOU2/EMS1/CARD/NET0 PCIE pci_32 primary OCC /SYS/IOU2/EMS1/CARD/SCSI PCIE pci_32 primary OCC /SYS/IOU2/PCIE2 PCIE pci_33 mars OCC /SYS/IOU2/PCIE5 PCIE pci_34 mars EMP /SYS/IOU2/PCIE8 PCIE pci_35 primary EMP /SYS/IOU2/PCIE11 PCIE pci_36 primary EMP /SYS/IOU2/EMS3/CARD/NET0 PCIE pci_36 primary EMP /SYS/IOU2/EMS3/CARD/SCSI PCIE pci_36 primary EMP 12 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Preparing for Hotplug Check in “mars” using the hotplug command root@mars # hotplug list -c Connection State Description __________________________________________________ IOU2-PCIE2 ENABLED PCIe-Native IOU2-PCIE5 EMPTY PCIe-Native 13 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Preparing for Hotplug A Detailed view on PCIE2 root@host-9-78 # hotplug list -clv IOU2-PCIE2 Connection State Description Path _____________________________________________________ IOU2-PCIE2 ENABLED PCIe-Native /pci@b40/pci@1/pci@0/pci@2 Device Usage ________________________________________________ SUNW,qlc@0 - fp - disk - fp@0,0 - SUNW,qlc@0,1 - fp - disk - fp@0,0 - 14 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Removing the HBA root@mars # hotplug poweroff IOU2-PCIE2 root@mars # hotplug list -c Connection State Description __________________________________________________ IOU2-PCIE2 PRESENT PCIe-Native IOU2-PCIE5 EMPTY PCIe-Native 15 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Re-Inserting the HBA root@mars # hotplug poweron IOU2-PCIE2 root@mars # hotplug list -c Connection State Description __________________________________________________ IOU2-PCIE2 POWERED PCIe-Native IOU2-PCIE5 EMPTY PCIe-Native root@mars # hotplug enable IOU2-PCIE2 root@mars # hotplug list -c IOU2-PCIE2 Connection State Description __________________________________________________ IOU2-PCIE2 ENABLED PCIe-Native IOU2-PCIE5 EMPTY PCIe-Native 16 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 17 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. SPARC M6: Processor Overview PCIe . SerDes MCU SerDes 12 SPARC S3 cores, 96 threads MIO . 48MB shared L3 cache L3 SPARC L3 Core Coherence & ScalabilityCoherence . 4 DDR3 schedulers, maximum of 1TB of memory per socket SPARC SPARC L3 SPARC L3 SPARC SPARC Core Core Ctl Core Ctl Core Core . 2 PCIe 3.0 x8 lanes SerDes Crossbar . Up to 8 sockets glue-less scaling SPARC SPARC L3 SPARC L3 SPARC SPARC . Up to 32 sockets glued scaling Core Core Ctl Core Ctl Core Core . 4.1 Tbps total link bandwidth SPARC SerDes L3 Core L3 . 4.27 billion transistors SerDes MCU SerDes PCIe 18 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Sun 4/260 „Sunrise“ July 1987 . MB86900 chip set implementing SPARC V7 ISA – First implementation of SPARC – Two 20.000-gate 1.2µm CMOS Gate Arrays . Manufactured by Fujitsu Limited . MB86900 Microprocessor . MB86910 Floating-Point Unit – 16.67 MHz, 10 MIPS 19 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 25+ Years of SPARC Processors 1987 1988 1992 1995 1996 2000 2002 2005 2007 2010 2011 2013 Sunrise: 1st SPARC Processor UltraSPARC IV+ UltraSPARC II SPARC T5 SuperSPARC I UltraSPARC IIIi UltraSPARC T2 SPARC T4 SUNRAY SUNRAY UltraSPARC I UltraSPARC III UltraSPARC T1 UltraSPARC T3 SPARC M5 20 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. SPARC M6: Processor Overview PCIe . SerDes MCU SerDes 12 SPARC S3 cores, 96 threads MIO . 48MB shared L3 cache L3 SPARC L3 Core Coherence & ScalabilityCoherence . 4 DDR3 schedulers, maximum of 1TB of memory per socket SPARC SPARC L3 SPARC L3 SPARC SPARC Core Core Ctl Core Ctl Core Core . 2 PCIe 3.0 x8 lanes SerDes Crossbar . Up to 8 sockets glue-less scaling SPARC SPARC L3 SPARC L3 SPARC SPARC . Up to 32 sockets glued scaling Core Core Ctl Core Ctl Core Core . 4.1 Tbps total link bandwidth SPARC SerDes L3 Core L3 . 4.27 billion transistors SerDes MCU SerDes PCIe 21 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. SPARC M6 CPU Block Diagram DDR3 – 1066 MHz DDR3 – 1066 MHz DDR3 – 1066 MHz DDR3 – 1066 MHz Scalability Links 12 Gbps per lane BoB BoB BoB BoB BoB BoB BoB BoB 16 GBps eachdirectionGBps 16 – 4 lanes per link GBps x PCIe 3.0@ 8 28 (144 Gbps) Coherency Links SLC Memory Memory IO 12.8 Gbps per lane Control Control Subsystem - 12 lanes per link SLC (1075 Gbps) Coherence Unit Coherence Unit SLC Link 0 Link 1 SPARC SLC S3 Core SLC Link 2 128 KB L2$ Link 3 L3$ B0 L3$ B2 L3$ B1 L3$ B3 SLC L3$ B0 16 KB L1I$ 12MB1MB,16,16--wayway 12MB,16-way 12MB,16-way 12MB,16-way Link 4 16 KB L1D$ 12 x 5 Crossbar (~620GBps bandwidth) Link 5 FGU C6 C7 C8 C9 C10 C11 Switch 2x4 Coherency C0 C1 C2 C3 C4 C5 Link 6 Crypto 8 threads per Core 22 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. T5 System Interconnects 1-Way 2-Way 4-Way Single Socket Dual Socket 6-Way 8-Way 23 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 8 Socket Local Coherency & Data Interconnect All-to-All Interconnect Where the Node-to-Node Fabric is 12 diff pairs per link in each direction. DIMMS DIMMS M5/T5 M5/T5 T5 interconnect bandwidth= 157.5 GB/sec T5-8 interconnect bandwidth= 1260 GB/sec DIMMS M5/T5 M5/T5 DIMMS PCIe Gen3 Bandwidth 8 diff pairs per ports At 8Gb/sec ~8GB/sec/direction POINT-TO-POINT LOCAL ~8Gb/sec/lane X 8 lanes = 64 Gb/s INTERCONNECT X 2 directions X 2 Ports/chip DIMMS = 256Gb/s/chip M5/T5 M5/T5 DIMMS = 32 GB/s per chip DDR3-1066 Memory Bandwidth DIMMS T5 is 133 GB/sec DIMMS M5/T5 M5/T5 T5-8 is 1064 GB/s 24 Copyright © 2013, Oracle and/or its affiliates.