®
Hyperscale Computing, Fabrics and the Datacenter of Tomorrow
Larry Wikelius
® 1 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
How Does the Industry Define Hyperscale Today? q Webopedia.com: Hyperscale computing refers to the infrastructure and provisioning needed in distributed computing environments for effectively scaling from several servers to thousands of servers. q Whatis.techtarget.com: Hyperscale computing is a distributed computing environment in which the volume of data and the demand for certain types of workload can increase exponentially yet still be accommodated quickly in a cost-effective manner. q Wikipedia: Did you mean: Hyperspace? q Wordnik.com: Sorry – no definition found
® 2 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
http://www.hyperscale.com/
® 3 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Overview
l Hardware Technology Trends Leading the Way
l System-on-Chip
l Fabric Technology
l Software Defined Networking
l Software Technology Trends that Enable and Drive
l I/O Driven Workloads
l Big Data
l Impact upon the Changing Datacenter
l Where are we headed…
® 4 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Technology Trends – Single Core Performance We are reaching single-core performance limits -> 52% historical growth slows to 22% <-
th ® Source: Computer Architecture 5 edition 5 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Historical Computing Efficiency Efficiency doubled every 1.57 years from 1946 - 2009
Source: Implications of Historical Trends in the Electrical Efficiency of Computing, 2011 ® 6 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Technology Trends – Mobile Technology
q Mobile processors designed for energy efficiency q Target good performance/energy use trade-off q Pioneered System-on-Chip (SOC) integra on technology q Deep on-chip power/clock/frequency ga ng q Modern mobile devices also require performance q Tablets replacing laptops and desktops q Mul -core cellphones, with high end games q Mixture of asynchronous compute w/offload q Energy aware scheduling/power decisions
® 7 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
® 8 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Technology Trends – System-on-Chip (SOC)
q Originally an embedded systems concept q Allows very high density solu ons (e.g. cell phones) q Integrate customized value q Standard compute (ARM core) with customized devices q Specialized peripherals, offload engines, management q Package-on-Package allows stacking q Stack RAM/flash on top of the SoC q En re system in a few inches
® 9 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Technology Trends – System-on-Chip (SOC)
Calxeda EnergyCore™ EnergyCore™ SOC Processor Complex Management Processor Complex Multi-core ARM® processors Engine integrated with high Advanced system, power bandwidth memory EnergyCore ARM controllers and fabric optimization Management and management for ARM Engine energy-proportional ARM Memory
computing Controllers L2 Cache L2Cache ARM EnergyCore™ Fabric Switch I/O Controllers I/O Controllers Standard drivers, SATA, PCIe, EnergyCore Integrated high-performance standard interfaces. No Ethernet, SD/ Fabric Switch fabric converges internode surprises. eMMC communication, I/O, and storage networking
® 10 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
System Benefits of True SOC Integration Reduced Design Complexity
4 GB DRAM ECC mini-DIMMS Quad-core servers 4 SATA / Node Power, SATA, & Fabric
Approximately 10”
q A cluster of 4 servers q 20 wa s q No cables
Image: Calxeda EnergyCard powered by Calxeda EnergyCore (source: Calxeda.com)
® 11 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Technology Trends – Integrated Management
q Future datacenters will have millions of server nodes q None of these will have displays or serial cables q None of these will use exis ng “KVM” interface q Each SoC can integrate management features q Out-of-band IPMI, secure provisioning, etc. q Value-add opportuni es for OEMs/vendors q Provisioning will eventually move to an injec on-model q No thundering herd of servers hi ng PXE/DHCP q Inject (push) the OS/system images
® 12 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Fabric connectivity
® 13 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Technology Trends – Fabrics q Transparent, reconfigurable interconnect between many SoCs q No physical network ports to plug between systems q Redundant paths and rou ng op miza ons built-in q Cluster level, server level and/or policy level controls q Virtualized IO and networking to all nodes q Replaces “top of rack” switch/cabling mess q Obviates need for one-to-one storage q Lower power and higher density q Enables true So ware Defined Networks
® 14 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Fabric Topologies q Flexibility for market specific topology optimizations q North/south traffic (think web server) q East/west traffic (Fabric SANs, computational analytics) q Bisection bandwidth (classic HPC) q Appliances – Data flow optimized topologies q Different topologies spanning chassis
Dual Fat Tree
Butterfly Fat “Classic” Fat CLOS Network Tree 2D Torus Tree
® 15 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Technology Trends – Software Defined Networks (SDN)
q Decouple data and control panes q Move control into (flexible) so ware q U lize commodity general purpose compute
q Programmable network layers q Dynamic reconfigura on q Service innova on (service-driven networks) q Virtualized infrastructure
q OpenDaylight Project q Linux Founda on SDN project
® 16 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Technology Trends – I/O Driven Workloads
q Many modern workloads are dominated by I/O q Web Tier obvious example, but also data analy cs q Storage and networking requirements escala ng q Reduced requirement for big iron horsepower per-node q Performance in aggregate across sea of nodes q Connected through massive bandwidth fabrics q Focus on rack level throughput q Industry Benchmark example – Data Intensive Focus q Graph500 - h p://www.graph500.org/
® 17 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Technology Trends – Big Data
q Distributed applica on level storage becoming cri cal q Ceph/Gluster and other Open Source solu ons q Never throw any data away q MapReduce approach to processing huge data sets q “Map” large problems into smaller data chunks q “Reduce” results from many nodes into end result q Apache Hadoop is the well-known implementa on
® 18 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Technology Trends – IAAS - OpenStack
q Hyperscale provisioning requires novel technology q Integrated management of whole data center q Nodes frequently re-installed (workload dependent) q “Physicaliza on” par ally displaces virtualiza on q On-chip management leverage for thin provisioning q OpenStack becomes the “Datacenter OS” of the future
® 19 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Changing Data Center – Energy Efficiency
q Data center energy costs are rising q Power Usage Effec veness (PUE – lower numbers be er) q Performance-per-Wa as a key compute measure q Hard limits on per-rack power consump on q Prevent business as usual approach q Workarounds q Infrastructure equipment overhead (networking, etc.) q Also associated HVAC infrastructure q Carbon dioxide and climate concerns
® 20 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Changing Datacenter – Hyperscale
q Rethink conven onal datacenter blade approach q Integra on (SoC, Fabric, etc.) q Drama cally reduce energy consump on q 1000+ nodes possible in a rack today q 10’s of thousands within 3-5 years q Performance in aggregate, not per core q Many simple nodes replace single points of failure q Hyperscale designs leverage simpler hardware
® 21 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Changing Datacenter – Vertical Integration
q Emerging shi toward purpose-built ver cal designs q Leverage standard (ARM) compute cores q Integrate I/O and peripheral devices on SoC q Integrate applica on specific accelerators (offload) q Open Source principals applied to hardware design q Facebook OpenCompute provides reference designs q Leverage high-value ODMs
® 22 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Changing Datacenter – Fabrics
q The “real story” behind ongoing datacenter revolu on q Next big ba leground in server design q Fabric Wars? q Disaggrega on of resources q Separate Compute from RAM, and Storage q U lize dynamically configured million node fabrics q OpenCompute blazing a trail
® 23 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Where are we Headed…
q Million node systems become a reality q New opportuni es in Compute/Big Data/Analy cs q Disrup ve impact across the Data Center q Future datacenters are dark places nobody goes q Flood fill with SoC/Fabric designs q Failure-in-Place up to set threshold q 3-5 year disaggregated deprecia on
® 24 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
Closing – Now How do we Define Hyperscale? q Fine grain complete building blocks q “Infinite” HW resource – at a scale not previously seen q Solution Optimized q Connectivity and throughput drives implementation q Breakthrough in overall performance and performance efficiency across Data Center
® 25 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013
The Real Future for Energy Efficiency Computing
Image: An HP Redstone 32-bit ARM server under JCM pedal power at RH Summit
® 26 Hyperscale, Fabrics, and the Datacenter of Tomorrow | End User Summit 2013