white paper

SoftBank Tests Next-Generation IT Platform Based on Intel® Rack Scale Design in Pursuit of Information Revolution

Intel® Rack Scale Design for Data Center Efficiency

Executive Summary SoftBank Corp. is pursuing an “information revolution” that will spur the latest innovations in technol- ogy, including the of Things (IoT), artificial intelligence (AI), and smart robots. The company needs the IT platforms which support its rapidly growing businesses to have agility to optimize real-time performance and high efficiency to utilize all available resources. The consolidation of its Japanese companies is what led SoftBank to consider the Intel® Rack Scale De- sign, which pools resources at the rack level and allocates them via software. During workload-based testing, a development test demonstrated a potential reduction in the number of servers, a great reduction in staff workload, and a significant electrical conservation. The test result also showed “Intel® Rack Scale Design is an that both server and network delivery times and fault response times could be significantly reduced. Having concluded that Intel® Rack Scale Design is suitable as a next-generation platform for real-time integral piece of architecture for computing, SoftBank is now planning to test expansion functions and functional improvements in real-time computing at preparation for the launch of Intel® Rack Scale Design. SoftBank as we accelerate our Consolidation of Japanese to give them the competitive advantage and global strategy including the IoT, Telecommunications Companies differentiate themselves, they must adopt Poses Challenge for IT Infrastructure a common IT platform to achieve synergies artificial intelligence, Consolidation between the four merged companies and improve efficiency. and smart robots.” • Testing Background Another factor behind SoftBank’s desire to In April 2015, SoftBank Mobile, SoftBank BB, – Mr. Amane Kito consolidate its IT platforms is to create an “in- SoftBank Telecom, and Y!Mobile were con- Senior Vice President, CISO and formation revolution”, a policy they pursue in solidated into a single new company, SoftBank Head of Information Systems Unit recognition of the important role that the IoT, Corp., that handles both mobile and fixed SoftBank Corp. AI, and smart robots will play in future growth. line telecommunications. This created a new In the near future, network connections will be organizational structure under which the bulk commonplace not only for mobile devices but of SoftBank Group’s Japanese telecommuni- also other devices such as industrial equip- cation businesses are consolidated. ment, robots, and different type of sensors, Mr. Amane Kito One issue that comes with the structural with a forecast of 50 billion online devices in Senior Vice President, CISO and reorganization is how to consolidate the IT 2020. This could lead to the emergence of a Head of Information Systems Unit platforms. While the business activities of steady stream of new IoT-based businesses, and in turn, the potential to change the fun- Mr. Masahiro Sekiya the four old companies have merged, they Senior Director retain the same IT infrastructure as before. damental role of these Internet “things.” If AI- System Infrastructure Division Furthermore, the gradual acquisition of a integrated smart robots enter widespread use, Information Systems Division variety of other companies has created a silo the need to enable robots to interpret human structure. Among the server, storage, and emotions by using large amounts of data will Mr. Toshiyuki Sugiyama rise. Accordingly, there will be a demand for System Infrastructure Development Section network hardware from different vendors and faster and more scalable computing resources System Infrastructure Division operating under different standards were a Information Systems Division range of general-purpose systems. In order to support the real-time processing and vi- SoftBank Tests Next-Generation IT Platform Based on 2 Intel® Rack Scale Design in Pursuit of Information Revolution

sualization of the high volume of transactions Intel® Rack Scale Design is one of the archi- 5. Top-of-rack (ToR) switch arising from this proliferation of devices. tectures used to implement the concept of 6. Computing resources drawer software-defined infrastructure (SDI) being These are some of the considerations that • Each drawer was fitted with an Intel® Xeon promoted by Intel, which abstracts the vari- piqued SoftBank’s interest in Intel® Rack Scale or Intel® Atom™ processor module Design, a vertically integrated architecture for ous hardware used at a data center, including • Fabric switch with Intel® Ethernet Switch consolidating server, storage, and network processors, memory, storage, and networking, FM5224 silicon base hardware at the rack level and interconnect- and uses software to modify its setup. Use of ing them via high Ethernet Fabric. Intel® Rack Scale Design enables hardware • Each drawer and ToR switch connected Senior Vice President, CISO and Head of Infor- configurations to be modified as necessary. via 40GbE optical cable mation Systems Unit of SoftBank, Mr. Amane Whereas resources in the past were only 7. Intel® Xeon® E5 v3 processor modules and Kito, put it as follows. available after servers, storage, or other Intel® Atom™ C2000 SoC modules (each devices were added individually, with Intel® “Intel® Rack Scale Design has the potential module has 12 Intel® Atom™ C2000 SoCs) Rack Scale Design, they can be made available to dramatically improve the utilization of 8. JBOD storage(object storage) by software setting in much the same way as computing resources that in the past have selecting a “component” from a pool of avail- not been efficiently utilized. SoftBank Group • Testing dates able resources organized on the rack level. is currently working on a unified platform for From May to September 2015 (effective test- Mr. Sekiya commented, “Because an archi- real-time computing called “Chronos.” We ing time of 10 weeks) tecture like Intel® Rack Scale Design is such have been conducting tests on an Intel® Rack a groundbreaking development, this testing Scale Design development system because we • Tests will have considerable impact on our future IT expect it to play an important role in imple- SoftBank tested its basic performance and infrastructure strategy.” menting this new platform. functions by using the web UI for resource “One of SoftBank Group’s strengths is that Prototype Evaluation System Tested monitoring; composing servers from the com- we are quick to adopt advanced Information over an Effective 10-Week Period pute pool; assigning storage and networking, and Communication Technology(ICT), and installing and booting the OS on the server; ever since our formation we have consistently • Testing environment installing and executing the application; and testing the Intel® Rack Scale Design API. The been among the first to test promising new The system configuration used for testing following tests were conducted with an em- technologies. This attitude has allowed us to is listed below. The evaluation system was phasis on verifying whether the basic concept take the first steps toward implementation a prototype used by Intel for development, of Intel® Rack Scale Design had been achieved. of advanced ICT by using the only evaluation with hardware put together from off-the- system in , therefore learning about shelf components. [Test 1] Intel® Rack Scale Design pod Intel® Rack Scale Design before anyone else. 1. Intel® Rack Scale Design rack management function using Pod Manager One of our future goals is global deployment of this system as a common global platform • 26U rack Testing confirmed that Pod Manager, the con- for SoftBank Group.” 2. Rack was fitted with a common power sup- trol center for Intel® Rack Scale Design could ply detect key components, such as computing and storage on the rack and man- Confirming Functions Provided by Intel® 3. Rack was fitted with a common cooling age them centrally. It also confirmed when Rack Scale Design and Operation under system Actual Workloads computing resources were added or removed, 4. Pod Manager (also incorporating rack man- the change was automatically detected by the • Testing Objectives agement functions) system and the resource was recognized as The aim of testing was to affirm that the Intel® • A small desktop PC kit (Intel® NUC) available or unavailable. Rack Scale Design’s functions can optimize mounted at the bottom of the rack resources in real time. The first objective was Pod to confirm that the processors, memory, stor- Pod Manager by domain Pod Manager age, and network could be configured on the hardware platform. The second was to verify Rack the ability to execute the actual workload on TOR RMC by rack RMC top of the platform OS and middleware used Drawer

PCIe by SoftBank. This work also included testing FM6324 PCIe CPP network coordination and the optimization switch by drawer PSME of compute, storage, and other resources to PCIe

SMBus assess the effectiveness of the automation Xeon essential for the next-generation platform. module PCH BMC by module Xeon SOC Xeon SoC DMI BMC MMC BMC MMC Mr. Masahiro Sekiya, Senior Director, System MMC Xeon uServer

Infrastructure Division, Information Systems DDR DDR DDR E5 DDR DDR DDR DDR module Division, commented that, “The overriding ob- QPI jective of the testing was to confirm whether Avoton SOC DDR or not Intel® Rack Scale Design, as a next-gen- Xeon ••• DDR DDR DDR DDR DDR DDR DDR DDR E5 M.2 Drive Avoton Compute

eration architecture, was suitable in practice Blade Compute Avoton Blade Compute Avoton Blade Compute Avoton Blade Compute Avoton Blade Compute Avoton Blade Fig. 1 Hardware Configuration and for deployment as a platform, and could be M.2 Drive Architecture of Evaluation System put to beneficial use.” SoftBank Tests Next-Generation IT Platform Based on 3 Intel® Rack Scale Design in Pursuit of Information Revolution

[Test 2] Assignment from resource pool The testing demonstrated the ability to per- using Pooled System Management Engine form provisioning using the configuration they Intel® Rack Scale Design Pooled System Management Engine is a man- set up, including networking, with resources agement tool for drawers. Testing confirmed in the rack being detected automatically, and Intel® Rack Scale Design is a reference that it could be used to allocate resources the required computing resources and storage architecture devised as part of Intel’s appropriately. assigned by software alone using the Web UI. work on its future data center strategy The testing represents a major step forward (software-defined infrastructure). A [Test 3] Intra-pod storage management toward SDI, as Mr. Sekiya noted by saying, logical architecture that subdivides function and allocation through integration “What is significant is that this is not virtu- processors, memory, storage, and with distributed block storage (Ceph) alization, but an actual physical, bare-metal networking to provide highly efficient Testing confirmed that storage disk capacity environment.” resource pools, it is designed to al- could be allocated appropriately for the dis- That trouble-free execution of the workload locate resources automatically and tributed storage (Ceph) used for testing. was achieved on a configuration in which dynamically in response to applica- the drawers had a mix of both Intel® Xeon® tion demand. Use of Intel® Rack Scale [Test 4] Division and allocation of network and Intel® Atom™ processor modules. This Design enables the implementation resources demonstrated Intel® Rack Scale Design’s re- of a software-controlled data center Testing confirmed that the pool of network re- sponsiveness in permitting the reallocation of without a large investment. Specifical- sources on the rack could be correctly divided processor resources during use, even in mixed ly, the processors, memory, storage, up and allocated to the selected nodes. configurations that, in the future, may also in- and networking in a rack are designed clude even faster programmable LSIs (FPGAs) for flexible allocation and pooled via [Test 5] Efficiency of cabling using optical and coprocessors (such as the Intel® Xeon Phi™ a fabric switch. Rack drawers can hold cable connection from networking product family). multiple boards on which proces- integrated into drawer The current testing only went as far as the sors, memory, and networking are In a typical Intel® Rack Scale Design Plat- manual allocation of resources. In the future, mounted. Processors are connected to form, the compute nodes within a drawer are the intention is to test a fully automatic the switch in the drawer, while the top- connected to a disaggregated switch which mechanism for maintaining capacity without of-rack (ToR) switch (so called because then connects to a TOR. This allows fewer any human intervention whatsoever. The it is located at the top of the rack) cables and prevents the tangle of cables that software should identify system overload- connects to these drawer switches might otherwise be required. (Note: Based on ing automatically and respond by adding just allowing fewer cables. disaggregated switch cost and performance the right amount of additional resources, and requirements, some systems may decide to likewise, conserve resources when they are connect from compute node to TOR directly.) no longer needed. This should lead to higher Testing confirmed that when a server was resource utilization and further optimization of 3. Review of Testing setup for testing, a network connection and IP IT infrastructure. address were automatically provisioned. Testing of Intel® Rack Scale Design was a 2. Improved operational efficiency series of new discoveries. Looking back at [Test 6] Run and test actual workloads Testing demonstrated that Intel® Rack Scale the work, Mr. Toshiyuki Sugiyama of System The same OS and middleware as used on the Design significantly reduced administration Infrastructure Development Section, System actual IT platforms were installed and applica- workloads by pooling resources in a rack and Infrastructure Division, Information Systems tions run to conduct workload testing to verify managing them centrally using Pod Manager. Division commented that, “This was the first that Intel® Rack Scale Design could be used in Whereas adding processors or memory under time I had ever been involved in building IT SoftBank’s commercial systems. The installed previous administrative environments, for infrastructure from scratch, and while it was OS was an Ubuntu* Linux* distribution and the example, required servers to be purchased a big challenge to test actual operation in middleware was a scalable key-value store (KVS) and added one at a time, this can be accom- parallel with getting the prototype up and for column-based data storage. Testing con- plished with Intel® Rack Scale Design by add- running, it left me with a sense of achieve- firmed the ability to run the KVS on Ubuntu* and ing the required processor modules, memory, ment.” For SoftBank, this work on a new ar- that the workload could be executed on the KVS. or storage to a computing resources drawer chitecture also represented an opportunity in the rack. This was found to be a much to gain a major advantage. more efficient means of operation, and also [Test 7] Intel® Rack Scale Design Mr. Sekiya commented that, “The arrival of eliminated the need to physically reconnect API testing a new technology constitutes a change in network cables. This involved evaluating the standard API pro- the world, and success or failure is deter- vided with Intel® Rack Scale Design. The testing Routine resource monitoring has previously mined by how quickly you can gain experi- provided feedback on APIs that SoftBank would involved individual monitoring of physical ence from it. We were able to advance our like added for use in commercial systems. server resources, but resource optimization IT strategy by being the first to work with a is impossible when physical servers are inde- technology that has the potential to become Confirmation of Basic Operation in pendent of each other, and can be influenced a standard some years in the future, and to Preparation for Fully Automated by workloads. Changing to the rack-based find out what it is capable of. This testing Operation in Future Intel® Rack Scale Design makes it possible to should represent a major turning point for centrally manage the physical environment. SoftBank. It seems significant to me that this • Testing results It was also found that maintenance could be new technology was offered not by a manu- 1. Testing of basic functions for software- streamlined by conducting it at the compo- facturer or system integrator but by Intel, a defined infrastructure (SDI) nent rather than the server level. device vendor.” SoftBank Tests Next-Generation IT Platform Based on Intel® Rack Scale Design in Pursuit of Information Revolution

Confirmation of Cost and Workload force for accelerating SoftBank Group’s “For SoftBank, Intel® Rack Scale Reductions strategy of halving non-personnel costs and achieving a two-fold increase in productivity. Design is a potential opportunity • Remarks for changing the world.” The test results demonstrated benefits Testing of Expansion Functions in terms of both cost and administration. and Functional Improvements – Mr. Masahiro Sekiya Results indicated a potential reduction of in Preparation for Commercial Senior Director, servers compared to past architectures. Deployment of Intel® Rack Scale Design System Infrastructure Division Along with this, the company believes it will Systems Information Systems Division be possible to expect a significant conserva- tion of electric power and great reduction • Future Activities in operational staff workload. In terms of As the testing was conducted using a administration, tests indicated that required pre-release prototype, it did not extend to point that, “A world in which machine learn- resources could be provided more quickly, considerations of actual operation. How- ing, image recognition, and voice recognition with greatly shortened delivery times of ever, based on the test results, the intention are performed in real time will need proces- servers and networking, while dealing with is to use a commercial Intel® Rack Scale sors, I/O, and other technology with higher faults promptly. Design system, once available, to conduct performance. Accordingly, we anticipate the Mr. Kito commented on these results by further testing on expansion functions for integration of Intel® Rack Scale Design with saying, “As the scale of SoftBank’s business use in commercial operation, integration new types of processors, such as FPGAs and is expanding to encompass the IoT, AI, and with OpenStack, and testing of functional co-processors, and also advanced memory smart robots, the ability of Intel® Rack Scale improvements. The aim of testing on a com- and storage technologies that use next- Design to implement automation with high mercial Intel® Rack Scale Design system will generation non-volatile memory. We also performance and achieve a major reduction be to execute the applications used by the recognize its potential as a next-generation in IT costs not only facilitates faster decision business, not just the workload execution IT platform that will support the global making, it also has the potential to become testing conducted this time. Furthermore, strategy of the SoftBank Group, including in an important platform for growth.” Further- with a view to utilizing future new technolo- operational terms and through the testing of more, IT is seen as providing the driving gies released by Intel, Mr. Kito made the the latest technology.”

Expectation for Intel® Rack Scale Design Business Scale IT Cost

Growing businesses Using Existing Architecture +IoT +AI Cost +Robot Reduction +Contents

Using Intel® Rack Scale Design - High Performance Mr. Amane Kito Mr. Masahiro Sekiya Mr. Toshiyuki Sugiyama - Automation Senior Vice President, CISO Senior Director, System Infrastructure Devel- - Optimization and System Infrastructure opment Section Head of Information Systems Division System Infrastructure Fig. 2 (Anticipated) Benefits of Adoption Unit Information Systems Division Division Information Systems Division Source: SoftBank Group Corp. Please note that the attribution and position are confirmed at the time of customer interview.

Intel does not control or audit the design or implementation of third party benchmarks or Web sites referenced in this document. Intel encourages all of its customers to visit the referenced Web sites or others where similar performance benchmarks or data are reported and confirm whether the referenced benchmarks or data are accurate and reflect performance of systems available for purchase. All dates and products specified are for planning purposes only and are subject to change without notice. Information in this document is provided in connection with Intel® products. No license, express or implied, by estoppel or otherwise, to any intellectual property rights is granted by this document. Except as provided in Intel’s terms and conditions of sale for such products, Intel assumes no liability whatsoever, and Intel disclaims any express or implied warranty, relating to sale and/or use of Intel products including liability or warranties relating to fitness for a particular purpose, merchantability, or infringement of any patent, copyright or other intellectual property right. Intel products are not intended for use in medical, life-saving, or life-sustaining applications. Intel may make changes to specifications and product descriptions at any time, without notice. Intel, the Intel logo, Intel Atom, Intel Atom Inside, Xeon, Xeon Phi, and Xeon Inside are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. SoftBank and the SoftBank name and logo are registered trademarks or trademarks of SoftBank Group Corporation in Japan and other countries. Copyright © 2016 Intel Corporation. All rights reserved. * Other company or product names are names, trademarks, or registered trademarks of their respective companies. JPN/1609/PDF/SE/BDG/YS 334307-002US