BOOTSTORM AND VM DISK ACCESS: NVMe vs. SATA Technical Report Antonio Cisternino, Maurizio Davini The University of Pisa, IT Centre Abstract As solid state drive (SSD) technology matures, the performance it delivers is constrained by old disk controller technologies. The NVMe protocol on PCIe interface allows you to bypass the controller by attaching a disk drive directly to the bus. In this report we present the results of comparing NVMe and SATA drive performance in the context of virtualization. In the experiment we observed how different kinds of drive react to a VM bootstorm and to intense simultaneous drive access from all the running VMs. Results show that an NVMe interface offers significant benefits over SATA: bootstorm time is on average 75% slower when using SATA; moreover, when all the VMs perform disk access using diskspd simultaneously NVMe outperforms SATA by a factor greater than four, offering on average more than 250MB/s bandwidth to each VM. Antonio Cisternino, Maurizio Davini TR-03-16 Executive Summary NVMe SSD drives provide more performance in terms of bandwidth and lower latency than SAS and SATA based counterparts (see [5]). However, it is always difficult understanding whether this increased performance will correspond to a real benefit in a real-world application. In this paper we have focused our attention on two important aspects of virtualization: bootstorm and disk performance from a VM standpoint. We present the results of our experiments to test the performance of virtual machines running on Microsoft Hyper-V with virtual drives stored on PCIe® and SATA SSD drives (20nm MLC). The solid state technology used in both drives is the same, so the tests allow an appreciation of the effective difference that the interface makes in terms of performance. Performed tests aim at understanding the potential impact of adopting controller-less disks on a virtualization platform and consist of: • checking the impact of PCIe interface when a bootstorm1 occurs; • verifying the throughput that can be achieved concurrently by using the open source diskspd tool [7] on all the virtual machines running on a single hypervisor. The results have shown that employing PCIe drives with NVMe offer significant improvements in the virtualization scenarios examined. In particular, we observed a speed- up of 80% in the bootstorm of 50 virtual machines, but improvement already shows up when booting just 5 VMs simultaneously. Once the VMs finished the boot they started generating sequential and random access on the virtual disk drive using the diskspd tool [7]; we measured the average bandwidth provided by a PCIe drive and compared it with a SATA drive; results show that when 50VMs access the virtual drives, stored on the same SSD physical drive, random access on a PCIe interface delivers on average 273MB/s to every VM whereas a SATA interface is capable of providing only 90MB/s in the same condition. Also in this case we noticed significant benefits with a smaller number of virtual machines in execution. 1 i.e. when in certain conditions the system has to start a large number of virtual machines simultaneously NB: Other names and brands may be claimed as the property of others. Introduction This report is the third in a series dedicated to evaluating solid state drive (SSD) performances. In the first report [5] we compared the performance of SATA and NVMe PCIe SSD connections using Hammer DB, showing that the NVMe protocol used for attaching drives directly to the PCIe bus offers significant improvements with respect to SATA controller in terms of bandwidth and latency under a database-like workload. In the second report [6] we tested the impact of PCIe interface with respect to SATA when aggregating drives using different forms of software defined RAID: Microsoft Storage Spaces [9] and Intel RSTe [8]; experiments have shown superior software RAID performance of PCIe interface, capable of delivering up to 10GB/s when aggregating four drives using RAID0. We are interested in exploring how changing the hardware interface used for accessing a SSD drive affects the performance of a server acting as hypervisor for a number of virtual machines. We focused our attention on two important areas for virtualization: bootstorm (i.e. simultaneous boot of many virtual machines) and virtual disk benchmarking (i.e. all VMs performing a disk benchmark simultaneously). Tests have been measured by combining performance counters of the hypervisor host running Windows Server 2012 R2, and the output generated by each VM running diskspd tool [7]. We used Windows 10 Enterprise as a guest OS executed by virtual machines. We monitored several performance counters, including CPU, memory, power absorbed, disk throughput and disk queue length. Virtual drives have been tested for 40 seconds executing diskspd for generating sequential access with 128KB blocks for read and write, and random access with 4KB blocks for read and write. Measuring boot time is not easy since VMs are opaque for the hypervisor. We configured guest VMs for auto-logon and to execute a power shell script as a startup script: a file indicating the current date and time has been generated before starting the disk test procedure. We considered the boot of all VMs complete by taking the latest boot completion time as indicated by generated files. Virtual machines stored files containing the test results on a network shared on the hypervisor available on an internal virtual network defined on Hyper-v. We used a DHCP server to assign IP addresses to virtual machines, and the IP of every virtual machine has been used as an identifier. Since every VM test generates six files and seven additional files contain the beginning time and the performance counters recorded for the experiments, the experiment running 50 VMs generates 307 files. We analysed the 2536 files generated by repeating the experiment for different VM numbers and drive kinds using an F# script designed to parse text files and access the binary log format generated by the perfmon tool. IT Center Università di Pisa P.I.: 00286820501 C.F.: 80003670504 Largo Bruno Pontecorvo, 3, I-56127 Pisa Italy We monitored several performance counters, including CPU, memory, power absorbed, disk throughput and disk queue length. Virtual drives have been tested for 40 seconds executing diskspd for generating sequential access with 128KB blocks for read and write, and random access with 4KB blocks for read and write. Measuring boot time is not easy since VMs are opaque for the hypervisor. We configured guest VMs for auto- logon and to execute a power shell script as a startup script: a file indicating the current date and time has been generated before starting the disk test procedure. We considered the boot of all VMs complete by taking the latest boot completion time as indicated by generated files. Virtual machines stored files containing the test results on a network share on the hypervisor available on an internal virtual network defined on Hyper-v. We used a DHCP server to assign IP addresses to virtual machines, and the IP of every virtual machine has been used as an identifier. Since every VM test generates 6 files and 7 additional files contain the beginning time and the performance counters recorded for the experiments, the experiment running 50 VMs generates 307 files. We analyzed the 2536 files generated by repeating the experiment for different VM numbers and drive kind using an F# script designed to parse text files and access the binary log format generated by the perfmon tool. Boot storm analysis Boot storm happens when a hypervisor starts simultaneously a number of virtual machines. Operating system boot generates a significant amount of disk operations that may trash the performance leading to a very slow startBootstorm of virtual machines andAnalysis of the services delivered by them. A possible mitigation is a deferred power on of lessBootstorm critical happens virtual machines. when a hypervisor starts a number of virtual machines simultaneously. Operating system boot generates a significant amount of disk operations that may trash the performance, leading to a very slow start of virtual Previousmachines experiments and the services have delivered already by presented them. A possible the superiority mitigation isof a NVMe deferred interface power-on over of less SATA, critical so it virtual comes machines. without surprise that the boot duration is faster when VM virtual drives reside on NVMe drive. However, what the Previous experiments have already presented the superiority of NVMe interface over SATA, so it comes as no surprise that experimentthe boot duration has clearly is faster shown when VMis that virtual the drives benefits reside are on significant NVMe drive. starting However, from what the the boot experiment storm of has just clearly 5 virtual shown machines,is that the benefitswhere we are found significant, the boot starting time from on NVMethe bootstorm taking half of just time five of virtualthe SATA machines, drive. Withwhere 50 we virtualfound themachines boot time theon NVMeboot time taking is half251s the when time usingof the NVMeSATA drive. and 452sWith 50when virtual using machines SATA. theThe boot disk time bandwidth is 251s when usage using in the NVMe graph and 452s when using SATA. The disk bandwidth usage in the graph explains the benefits: after five VM the SATA drive reaches explainsits maximum the capacity,benefits: delaying after 5 theVM requests the SATA made drive by reaches booting VMs.its maximum capacity delaying the requests made by booting VMs. Boot duration (s) Disk bandwidth (bytes/s) 500 2500000000 400 2000000000 300 1500000000 NVMe - N NVMe Disk (B/s) 200 Time (s) SATA 1000000000 Bandwidth SATA - D 100 500000000 Disk (B/s) 0 0 1 2 3 4 5 6 7 8 9 10 20 30 40 50 1 2 3 4 5 6 7 8 9 1020304050 #VM #VM IT Center DuringDuring the bootstorm,boot storm we we monitored monitored power power absorbed, absorbed, memory memory and CPU and usage.