Whitepaper Resource-Pooled Server Trends Data Center Software Division

Tencent Explores Datacenter Resource-Pooling Using Intel® Rack Scale Architecture (Intel® RSA)

Traditional rack servers do not meet today’s datacenter needs. Tencent worked with Intel on a proof-of-concept involving Intel® RSA technology.

Tencent, a leading provider of • Datacenters globally consume Internet Services in China, recently over 100 GWh per year, a figure collaborated with Intel on a proof- industry analysts expect to of-concept to demonstrate that exceed 130 GWh by 2016.3 resource-pooling—even in the early • As measured by the power stages of development—could usage effectiveness (PUE) index bring better experience to users, defined by The Green Grid, reduce power consumption, and CRAC (computer room air yield measurable total cost of own- conditioning) units alone can ership (TCO) savings. consume as much as half of a This paper addresses the reasons datacenter’s power needs.4 for developing a resource-pooled “As a leading provider of Internet Over the last decade, Tencent has server technology standard; been involved in several efforts to Services in China, Tencent has a keen explains some of the challenges we interest in Intel® RSA technology. How improve datacenter design, face; offers insights on trends in the including a partnership with other resource-pooling can help us reduce development of this technology; datacenter total cost of ownership while Internet business leaders to create and presents some of the findings a rack-design specification, called simultaneously improving our from the Tencent proof-of-concept. customers’ user experience is a key Project Scorpio. Tencent focuses on question for our business.” Datacenter growth—in storage, maximizing datacenter perfor- power consumption, traffic, and mance and increasing resource uti- Herry Wang processing needs—is driving inno- lization; reducing hardware Principal Engineer and Architect vations in constructing more cost- acquisition, operations, and mainte- Tencent Holdings Limited efficient facilities. These efficien- nance costs; reducing datacenter cies are direly needed as datacen- TCO; and ultimately delivering a ters burst at the seams: better user experience to cus- tomers. • Ninety percent of the world’s data has been created in the last five years, and we add more than 2.5 quintillion bytes to the total every day.1 • Worldwide mobile data traffic— projected to grow at a compound annual rate of 57% for the next four years—is expected to reach a throughput of 24.3 exabytes per month by the year 2019.2 Tencent Explores Datacenter Resource-Pooling Using Intel® Rack Scale Architecture (Intel® RSA)

The shortcomings of rack servers hardware configurations. As a every two to three years, while result, server specialization has storage capacity has been doubling The rapid expansion of datacenters become the norm. Tencent’s data- at a rate of about five years. Mis- worldwide has been fueled by the center, for example, houses hun- aligned technology advances such explosive growth in social media dreds of thousands of rack servers, as these produce gaps in server and the commoditization of rela- more than 90% of which have been optimization. An all-in-one server, tively inexpensive rack servers. custom-provisioned for specific with equally provisioned resources, Ironically, that very same prolifera- workloads and purposes. will see one area of its resources tion of rack servers has also intro- become outdated more frequently duced new concerns. For example, This expansion of server types has than another area. This misalign- many servers are not optimally con- partially mitigated the inflexible ment of technologies makes it diffi- provisioning of traditional cookie- figured for their purposes, which cult to upgrade to more efficient can result in waste and inefficiency. cutter servers, but it has also led to processors, memory, and storage broader server diversity in the data- without unnecessarily discarding The “one size fits all” server config- center, which introduces new chal- still-useful resources. uration does not work in today’s lenges to server resource datacenter. In resource-intensive management, day-to-day mainte- Compounding this problem is the environments, a traditional rack nance, and overall datacenter oper- fact that businesses seldom replace server suffers from low operation ations. Each new custom server servers as often as they’d like—or efficiency and low deployment den- type introduces an additional layer as often as they should. The sity. Some negative side-effects of complexity to datacenter man- majority of organizations have old, include the following: agement and maintenance. less efficient servers. • In compute-dense applications, Moreover, past efforts to compen- Homogeneously provisioned unused memory slots, HDD sate for different application needs servers could ameliorate these (hard disk drive) slots, and and inflexible provisioning have run unavoidable mismatches in tech- expansion slots negatively into unsynchronized server compo- nology advancements by isolating affect computing density. nent lifecycles, which have also the technology to specific resource • In memory-dense applications, raised the TCO. For example, CPU pools. unused expansion slots and performance has been doubling HDD slots waste server “real estate” that could be used for more memory. Figure 1 Diversity of datacenter workloads. Most organizations customize servers according to workload needs. Plotting these workloads on a graph separating CPU/memory and input/output • In storage-dense applications, capacity shows the disparity of server provisioning required to optimize servers for specific workloads. CPUs and memory might be overprovisioned. As Figure 1 shows, the modern compute environment features a Enterprise applications variety of workloads, each with dif- High-performance computing ferent compute, storage, and I/O needs. With I/O intensity on the Graphics rendering horizontal axis and CPU and memory intensity on the vertical axis, it is difficult, perhaps impos- Edge routing sible, for a traditional server—with Cloud RAN its balanced configuration of com- puting, memory, and storage E-commerce Content delivery and gaming resources—to support the wide variety of applications encountered Small cell in a modern datacenter. Storage dedupe For years, Tencent and many other Low-end networking cloud service providers (CSPs) CPU and memory intensive worldwide have been customizing servers to perform specific types of Dedicated hosting Cold storage tasks from different workloads, pro- visioning the servers with different I/O intensive

2 Tencent Explores Datacenter Resource-Pooling Using Intel® Rack Scale Architecture (Intel® RSA)

To address these old and new chal- In other words, virtualization and multinode (m-to-n) resource alloca- lenges, we must devise a solution cloud services go part of the way, tion. Cloud services remain virtual- that allows us to intelligently provi- but they are no substitute for ized at the software level, but sion servers and manage the newly resource pooling. resource-pooled servers provide configured datacenter more effi- further virtualization capability at The first example in Figure 2 shows ciently. the hardware level. a 1-to-1 single-server environment, with various apps running on a These last two approaches are not Virtualization is not enough single operating system on one contradictory, but complementary. As with many large Internet service physical server. To increase With resource-pooled servers, we providers, Tencent’s datacenter capacity, you add more servers, but can continue to run virtualization currently features sophisticated vir- it is already obvious that this leads and cloud server software on logical tualization and cloud services, pro- to underutilization. servers and create virtual machines viding end-users with a better for end-users. This shrinks the In the 1-to-n virtualized server resource allocation hole left by experience and minimizing IT model (center), a single physical cloud services, improves the utiliza- expenses. With such an efficient server can divide its physical environment, why would an IT tion of hardware resources, and resources and allocate portions to ultimately reduces TCO. Compared department seek to pool its servers’ multiple virtual machines. This does resources? to the generic virtualization of a reduce idle resources, but it does single server into multiple nodes, Virtualization is a good midpoint not entirely eliminate inefficiencies, multiserver-to-multinode virtual- as different workloads require spe- phase in datacenter server optimi- ization provides even better utiliza- zation. With a firm footing in data- cially configured servers that virtu- tion of a pool of servers’ resources. centers today, virtualization and alization alone is not equipped to cloud implementations have handle. The biggest challenges in resource- pooling are interconnectivity/ improved resource utilization for Pooled resource technology (right) datacenter servers; however, CPUs latency, provisioning, and manage- allows a datacenter to allocate ment software. and memory in such environments resources with even greater effi- are still often underutilized, due to ciency by providing multiserver-to- the granularity of server resources.

Figure 2 Virtualization overdrive. In a one-to-one environment, every physical server is a unique node. Adding users or nodes means adding servers. A virtualized server environment allows more users or nodes using the same resources as a one-to-one environment. Only in a disaggregated server environment, however, does the datacenter approach full resource utilization.

Physical server (1-to-1) Virtualized server (1-to-n) Disaggregated server (m-to-n)

App App App App App App

Operating system OS OS OS VM VM VM

Virtualization Resource management software Storage Network Memory CPU

Storage Network Memory CPU

Storage Network Memory CPU pool pool pool pool

3 Tencent Explores Datacenter Resource-Pooling Using Intel® Rack Scale Architecture (Intel® RSA)

Interconnectivity and latency promising high-speed, low-latency This Intel-based photonic system solution. Using light to move huge allows data transfers as fast as The first challenge facing devel- amounts of data at very high speeds 1.6 Tbps (terabits per second) and opers of a resource-pooling stan- with extremely low power over a distances of up to 300 meters.5 dard is the means of connecting the thin optical fiber rather than using Following this successful demon- servers. Regardless of the protocol electrical signals over a copper chosen, the inherent transmission cable has the added benefit of stration, Intel submitted a design latency of transmitting the data to leaving a small footprint, which guide to the datacenter commu- nity, which provides an overview for various servers only complicates allows increased rack/interface the problem. density. implementing an intrarack optical interconnect scheme that uses MXC Each of the many interconnection After nearly a decade of research photonic connectors and protocols to choose from—PCIe*, and innovation to prove the viability embedded optical modules to Ethernet, SAS, fabric, etc.— has of photonics networking, in Sep- deliver high-speed data rates. advantages and disadvantages, and tember 2013 Intel demonstrated an as a result, industry leaders who are Intel® RSA-based system with high- Provisioning and management developing resource-pooled server speed photonics components— standards are still searching for the including ClearCurve* optical fiber A resource-pooling implementa- optimal solution. Using Ethernet as and the new MXC* optical con- tion requires a new software stack the fabric backbone appears to be a nector that Intel codeveloped with to manage resource discovery, cost-effective and versatile method. Corning. A core building block for resource allocation, bookkeeping, However, an open solution should photonic communication that will monitoring, etc. Intel® RSA pack- be protocol-neutral, as advances help define the way datacenters are ages a complete software stack to may occur to push another option built in the future, the protocol- provide resource-pooling, dis- to the forefront. neutral MXC connector offers high covery, configuration, and manage- To resolve the network latency speed with a long reach and a rela- ment all in one (Figure 4). issue on the OSI physical layer, the tively low cost. It may be used for Intel® RSA management software Intel® RSA SDV incorporates pho- network links throughout a data- not only provides an API to manage tonics (light) technology (Figure 3), a center, over existing technologies low-level pooled resources, it also as well as Terabit Ethernet.

Figure 3 Rack-level interconnectivity with photonics. Photonics is the best option at the OSI physical layer. Corning ClearCurve* optical fiber, constructed of a conventional fiber surrounded by nanostructured reflector cladding, has a 7.5 mm (5/16 in.) bend radius—compared to a 38 mm (1.5 in.) radius for conventional optical cable. It is far more flexible and compact, making it ideal for the tight confines of a datacenter. Combined with the protocol-neutral MXC* connector, which would replace the ubiquitous RJ45 connector, this interconnectivity solution will provide legacy networking and protocol support as well as an upgrade path to next generation datacenter terabit interconnectivity. On the protocol side, Ethernet, PCIe*, SAS, and fabric are all viable options.

Ethernet

PCIe

SAS

Fabric

Fiber optic cable MXC connector Various protocols

4 Tencent Explores Datacenter Resource-Pooling Using Intel® Rack Scale Architecture (Intel® RSA)

provides APIs exposed to middle- Resource-pooling specifications Resource-pooling allows a data- layer software offered by VMware center architect to gather key server or OpenStack. The pooled Intel® RSA is a server resource- resource components into different resources generate considerable pooling specification with a defined resource pools, such as a com- amounts of provisioning and man- architecture, a clear roadmap, and puting pool, a memory pool, a net- agement data. Traditional IPMI working prototypes—part of Intel’s work pool, and a storage pool, as interfaces and protocols, based on plan to rearchitect the datacenter shown in Figure 5). the I2C bus, can be used for data by treating compute, storage, After disaggregating processors, communication, but their low data memory, and networking as a storage, memory, and networking rate and inability to carry informa- whole. Although the standard is still resources, a datacenter adminis- tion render them incapable of in development, Intel released soft- trator can then flexibly assign those meeting resource-pooling require- ware development vehicles (SDVs) resources to meet the demands of ments. Intel, Hewlett-Packard, and to customers in early 2015, and the individual workloads. The pooled Dell have formed a joint venture, Intel® RSA design is on course to resources are recomposed into an named Redfish, which will address debut officially by 2017. “instance,” and the underlying oper- this issue, and Intel will integrate Figure 4 shows the basic concept. ating system and applications run the solution into the Intel® RSA soft- Intel® RSA pools compute, network, the same way they normally would ware architecture. fabric, and storage resources at the on a traditional server equipped Note: Intel® RSA does not alter the bottom of the stack, and uses a with the physical resources as if operation of business software from multirack “pod manager” module to they were on the same physical third-parties, such as Azure, VSphere, serve as the software/firmware that server. This architecture lets admin- or privately developed hypervisors exposes the hardware underneath istrators easily manage and scale and management stacks. A resource- to the orchestration layers above resources to reduce the total OPEX pooled server is only rearchitected in that manage and enforce policies. its hardware design; from the (operational expenditures) software perspective, resource- Note: Memory and networking are pooling is transparent. difficult to pool; Intel is still working on methods to pool these resources effectively.

Figure 4 Pooled resource management. In addition to the concepts of pooled, disaggregated hardware and interconnectivity solution that overcomes traditional datacenter latency issues, the third piece of the Intel® RSA puzzle is the resource management layer. The Intel® RSA API interoperates with the hardware and various orchestration partners via a pod manager application.

Orchestration partner solutions OpenStack* VMware* Microsoft*

RSA API Pod manager Power Fault Telemetry Discovery Boot Configuration

RSA API

Pooled system Storage RMM Pod fabric Operations/ Boot Switch NVM Operations Rack Distributed node mgmt. service configuration management management temp/power switch mgmt.

5 Tencent Explores Datacenter Resource-Pooling Using Intel® Rack Scale Architecture (Intel® RSA)

Another benefit of resource- Forum (SPMF)—to deliver a Tencent’s proof-of-concept pooling is improved failure common baseline for improved recovery. According to Tencent’s datacenter architecture and man- Tencent implemented a proof-of- internal statistic operation data, agement. concept (POC) in 2014 to demon- hard disk drives (HDD) account for strate the benefits of resource- Many industry analysts expect the highest failure rate of all their pooling, using a relatively simple these collaborative efforts will server components. While HDD “shared boot” exercise. As shown in coalesce to meet the expectations failure on a traditional server could Figure 6, the POC setup consisted of cloud/web-based IT profes- very likely result in lost data, pooled of 20 servers—with their internal sionals who seek scalable platform storage protects against such data hard disk drives removed—func- hardware management with loss via redundancy and fast migra- tioning as the “computing pool” and existing toolchains, as well as the tion. A failed HDD may be replaced/ one storage server acting as a sim- expectations of end-users who repaired at IT management’s conve- plified “storage pool”. By allocating want simple, modern, and secure nience. 20 compute nodes as the boot management of scalable platform drives, the storage pool was able to Under Intel’s vision, the disaggre- hardware. assemble up to 20 logical servers. gated rack architecture will also fea- Emerging datacenter architectures ture photonic-based high- Note: The maximum number of server will improve operational efficiency nodes allowed is limited by network bandwidth technology, which pro- through increased resource utiliza- bandwidth (10 Gbps in this POC), as vides farther reach using fewer tion and interoperability, and bring shown in Figure 7. On a network with cables and offers extreme power significantly increased perfor- higher throughput, the same quantity efficiency compared to today’s mance and lower total cost of own- of LUNs would be able to support copper-based interconnects. more server nodes. ership. Tencent and Intel have contributed To serve as the interconnect To support these claims, Intel and to similarly focused industry initia- between the two “pools”, Tencent Tencent partnered to conduct a tives—such as Project Scorpio, connected the diskless CPU servers proof-of-concept that would high- Redfish, the Distributed Manage- with the LUNs on the aggregated light resource-pooling in a data- ment Task Force (DMTF), and the single-server storage pool over a center environment. Scalable Platforms Management 10 Gbps Ethernet network using

Figure 5 Intel® RSA architecture. Different types of servers for different resource pools. under the Intel® RSA model, datacenter servers would be optimally provisioned according to the pool they will inhabit. In this example, a typical CPU sever—configured with multiple high-performance CPUs and limited or no storage capabilities—would inhabit the computing pool, while a typical storage server—with several large HDDs and light processing resources—would inhabit the storage pool. The Intel® RSA standard will treat differently configured servers appropriately.

Typical CPU server Typical memory server Typical storage server

Resource management software

CPU pool Memory pool Network I/O pool Storage pool

6 Tencent Explores Datacenter Resource-Pooling Using Intel® Rack Scale Architecture (Intel® RSA)

iSCSI. The compute servers then attached to the virtual, network- Figure 7 Storage node bandwidth. Tencent’s accessible drives, and booted POC network, with 20 server nodes, required a Spotlight on Tencent without needing any local storage. network bandwidth of at least 1200 Mbps. Tencent ran a simulated workload Storage node network bandwidth Tencent uses technology to on the 20 servers in the computing enrich the lives of Internet pool to demonstrate full capability, 1200 users. Every day, hundreds of and found performance to be com- 1000 millions of people communi- parable to a traditional one-to-one 800 cate, share experiences, con- 20-node environment. sume information, and seek 600 A shared boot scenario such as this entertainment through our is just one use case to show benefit 400 integrated platforms. Ten- from the resource-pooling concept. 200 cent’s diversified services Based on prior internal testing and MB/s include QQ, Weixin/ WeChat the actual results from this POC, Nodes 4 8 12 16 20 for communications; Qzone Intel and Tencent estimate that a for social networking; typical datacenter could reduce Moreover, this POC indicates that a QQ Game Platform for online hardware deployment by 6 to 10% datacenter could reduce its number games; QQ.com and Tencent operating under the exact same of SKUs by standardizing on fewer News for information; and workload. server “classes” for deployment to Tencent Video for video con- the various resource pools. tent. That does not take into account other projected savings from Also note that the storage pool in Tencent was founded in resource-pooling, such as reduced the Tencent POC employed global Shenzhen in 1998 and went OPEX, reduced administration staff, RAID design, a common feature in public on the Main Board of better resource utilization, and most datacenters and SAN farms. the Hong Kong Stock better workload matching, to name This allowed Tencent’s datacenter Exchange in 2004. The Com- a few. administrators to use RAID manage- pany is one of the constituent ment software (Figure 8) to monitor stocks of the Hang Seng Index. Tencent seeks to evolve with the Internet by Figure 6 POC setup. The Tencent proof-of-concept included a computing pool of 20 servers with HDDs investing in innovation, pro- removed and a storage pool consisting of a single server with its HDDs configured into 20 LUNs (logical viding a hospitable environ- unit numbers)—one for each server in the computing pool. These devices were connected via a ment for partners, and “network pool” consisting of a 10 Gbps Ethernet network using iSCSI. staying close to users. For more information, visit Computing pool Storage pool www.tencent.com/en-us. 20 servers (without HDD) 1 server

Ethernet Storage server switch

20 LUNs

7 Tencent Explores Datacenter Resource-Pooling Using Intel® Rack Scale Architecture (Intel® RSA)

the physical drives in the storage Summary For more information on Redfish and pool. If a HDD failure were to occur, the Intel® RSA standard, visit: there would be no impact on the Resource pooling brings flexible http://www.dmtf.org/standards/redfish service layer, as the data would be and elastic deployment, component mirrored across several other phys- redundancy capability, better Learn how the Intel® DCM SDK can help resource utilization, and lower TCO. you address real-time power and ical drives. Additionally, datacenter thermal monitoring issues in your data administrators would be able to fix Although much remains to be explored, Tencent’s POC demon- center at software.intel.com/ this type of resource failure with datacentermanager. software, rendering 24/7 physical strated clear advantages of presence of onsite engineering sup- resource-pooling and highlighted Find a business solution that is right for port unnecessary. In this type of its value in cloud computing. your company. Contact your Intel environment, whenever the admin- Tencent will continue efforts to representative or visit the Reference istration staff does decide to exploit the newest computing tech- Room at intel.com/references. replace failed units, physical HDD nology and to work with partners to hotswapping can take place at the arrive at a rack design that would IT department’s convenience. provide a highly competitive server Other cost-saving benefits include solution for Tencent’s customers. independent HW/SW upgrades, Tencent will also continue to lead simplified administration, interop- and contribute to the Scorpio 3.0 erable solutions, and modularity. project, to ensure the industry will share this technology’s benefit.

Figure 8 HDD failure in a fault-tolerant RAID volume. RAID logical volume management software allows Tencent datacenter administrators to identify HDD failures and repair/replace faulty drives at regularly scheduled maintenance visits, rather than at the time of failure.

Endnotes 1. IBM estimates, 2013. 4. SubZero Engineering. http://www1.eere.energy.gov/ 2. “Cisco Visual Networking Index: Global Mobile Data femp/pdfs/ns/fupwg_sandiego_mainers.pdf. Traffic Forecast Update 2014–2019 White Paper”. Cisco 5. “MXC* Connectors Will Light Up 21st Century Systems. February 3, 2015. http://www.cisco.com/c/en/ Datacenters”. Mario Paniccia. September 2013. http:// us/solutions/collateral/service-provider/visual- www.intel.com/content/www/us/en/data-center/mxc- networking-index-vni/white_paper_c11-520862.html. blog-paniccia.html. 3. Based on Forest & Sullivan market survey.

Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Cost reduction scenarios described are intended as examples of how a given Intel- based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at www.intel.com. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725 or by visiting Intel’s website at http://www.intel.com/design/literature.htm. Intel and the Intel logo are trademarks of Intel Corporation in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2015 Tencent Holdings and Intel Corporation. All rights reserved. Please Recycle 332866-001