BOOTSTORM AND VM DISK ACCESS: NVMe vs. SATA Technical Report
Antonio Cisternino, Maurizio Davini The University of Pisa, IT Centre Abstract
As solid state drive (SSD) technology matures, the performance it delivers is constrained by old disk controller technologies. The NVMe protocol on PCIe interface allows you to bypass the controller by attaching a disk drive directly to the bus. In this report we present the results of comparing NVMe and SATA drive performance in the context of virtualization. In the experiment we observed how different kinds of drive react to a VM bootstorm and to intense simultaneous drive access from all the running VMs.
Results show that an NVMe interface offers significant benefits over SATA: bootstorm time is on average 75% slower when using SATA; moreover, when all the VMs perform disk access using diskspd simultaneously NVMe outperforms SATA by a factor greater than four, offering on average more than 250MB/s bandwidth to each VM.
Antonio Cisternino, Maurizio Davini
TR-03-16 Executive Summary
NVMe SSD drives provide more performance in terms of bandwidth and lower latency than SAS and SATA based counterparts (see [5]). However, it is always difficult understanding whether this increased performance will correspond to a real benefit in a real-world application. In this paper we have focused our attention on two important aspects of virtualization: bootstorm and disk performance from a VM standpoint. We present the results of our experiments to test the performance of virtual machines running on Microsoft Hyper-V with virtual drives stored on PCIe® and SATA SSD drives (20nm MLC). The solid state technology used in both drives is the same, so the tests allow an appreciation of the effective difference that the interface makes in terms of performance. Performed tests aim at understanding the potential impact of adopting controller-less disks on a virtualization platform and consist of:
• checking the impact of PCIe interface when a bootstorm1 occurs; • verifying the throughput that can be achieved concurrently by using the open source diskspd tool [7] on all the virtual machines running on a single hypervisor.
The results have shown that employing PCIe drives with NVMe offer significant improvements in the virtualization scenarios examined. In particular, we observed a speed- up of 80% in the bootstorm of 50 virtual machines, but improvement already shows up when booting just 5 VMs simultaneously. Once the VMs finished the boot they started generating sequential and random access on the virtual disk drive using the diskspd tool [7]; we measured the average bandwidth provided by a PCIe drive and compared it with a SATA drive; results show that when 50VMs access the virtual drives, stored on the same SSD physical drive, random access on a PCIe interface delivers on average 273MB/s to every VM whereas a SATA interface is capable of providing only 90MB/s in the same condition. Also in this case we noticed significant benefits with a smaller number of virtual machines in execution.
1 i.e. when in certain conditions the system has to start a large number of virtual machines simultaneously NB: Other names and brands may be claimed as the property of others. Introduction
This report is the third in a series dedicated to evaluating solid state drive (SSD) performances. In the first report [5] we compared the performance of SATA and NVMe PCIe SSD connections using Hammer DB, showing that the NVMe protocol used for attaching drives directly to the PCIe bus offers significant improvements with respect to SATA controller in terms of bandwidth and latency under a database-like workload.
In the second report [6] we tested the impact of PCIe interface with respect to SATA when aggregating drives using different forms of software defined RAID: Microsoft Storage Spaces [9] and Intel RSTe [8]; experiments have shown superior software RAID performance of PCIe interface, capable of delivering up to 10GB/s when aggregating four drives using RAID0.
We are interested in exploring how changing the hardware interface used for accessing a SSD drive affects the performance of a server acting as hypervisor for a number of virtual machines. We focused our attention on two important areas for virtualization: bootstorm (i.e. simultaneous boot of many virtual machines) and virtual disk benchmarking (i.e. all VMs performing a disk benchmark simultaneously). Tests have been measured by combining performance counters of the hypervisor host running Windows Server 2012 R2, and the output generated by each VM running diskspd tool [7]. We used Windows 10 Enterprise as a guest OS executed by virtual machines.
We monitored several performance counters, including CPU, memory, power absorbed, disk throughput and disk queue length. Virtual drives have been tested for 40 seconds executing diskspd for generating sequential access with 128KB blocks for read and write, and random access with 4KB blocks for read and write.
Measuring boot time is not easy since VMs are opaque for the hypervisor. We configured guest VMs for auto-logon and to execute a power shell script as a startup script: a file indicating the current date and time has been generated before starting the disk test procedure. We considered the boot of all VMs complete by taking the latest boot completion time as indicated by generated files.
Virtual machines stored files containing the test results on a network shared on the hypervisor available on an internal virtual network defined on Hyper-v. We used a DHCP server to assign IP addresses to virtual machines, and the IP of every virtual machine has been used as an identifier.
Since every VM test generates six files and seven additional files contain the beginning time and the performance counters recorded for the experiments, the experiment running 50 VMs generates 307 files. We analysed the 2536 files generated by repeating the experiment for different VM numbers and drive kinds using an F# script designed to parse text files and access the binary log format generated by the perfmon tool. IT Center Università di Pisa P.I.: 00286820501 C.F.: 80003670504 Largo Bruno Pontecorvo, 3, I-56127 Pisa Italy
We monitored several performance counters, including CPU, memory, power absorbed, disk throughput and disk queue length. Virtual drives have been tested for 40 seconds executing diskspd for generating sequential access with 128KB blocks for read and write, and random access with 4KB blocks for read and write.
Measuring boot time is not easy since VMs are opaque for the hypervisor. We configured guest VMs for auto- logon and to execute a power shell script as a startup script: a file indicating the current date and time has been generated before starting the disk test procedure. We considered the boot of all VMs complete by taking the latest boot completion time as indicated by generated files.
Virtual machines stored files containing the test results on a network share on the hypervisor available on an internal virtual network defined on Hyper-v. We used a DHCP server to assign IP addresses to virtual machines, and the IP of every virtual machine has been used as an identifier.
Since every VM test generates 6 files and 7 additional files contain the beginning time and the performance counters recorded for the experiments, the experiment running 50 VMs generates 307 files. We analyzed the 2536 files generated by repeating the experiment for different VM numbers and drive kind using an F# script designed to parse text files and access the binary log format generated by the perfmon tool.
Boot storm analysis
Boot storm happens when a hypervisor starts simultaneously a number of virtual machines. Operating system boot generates a significant amount of disk operations that may trash the performance leading to a very slow startBootstorm of virtual machines andAnalysis of the services delivered by them. A possible mitigation is a deferred power on of lessBootstorm critical happens virtual machines. when a hypervisor starts a number of virtual machines simultaneously. Operating system boot generates a significant amount of disk operations that may trash the performance, leading to a very slow start of virtual Previousmachines experiments and the services have delivered already by presented them. A possible the superiority mitigation isof a NVMe deferred interface power-on over of less SATA, critical so it virtual comes machines. without surprise that the boot duration is faster when VM virtual drives reside on NVMe drive. However, what the Previous experiments have already presented the superiority of NVMe interface over SATA, so it comes as no surprise that experimentthe boot duration has clearly is faster shown when VMis that virtual the drives benefits reside are on significant NVMe drive. starting However, from what the the boot experiment storm of has just clearly 5 virtual shown machines,is that the benefitswhere we are found significant, the boot starting time from on NVMethe bootstorm taking half of just time five of virtualthe SATA machines, drive. Withwhere 50 we virtualfound themachines boot time theon NVMeboot time taking is half251s the when time usingof the NVMeSATA drive. and 452sWith 50when virtual using machines SATA. theThe boot disk time bandwidth is 251s when usage using in the NVMe graph and 452s when using SATA. The disk bandwidth usage in the graph explains the benefits: after five VM the SATA drive reaches explainsits maximum the capacity,benefits: delaying after 5 theVM requests the SATA made drive by reaches booting VMs.its maximum capacity delaying the requests made by booting VMs.
Boot duration (s) Disk bandwidth (bytes/s)
500 2500000000
400 2000000000
300 1500000000 NVMe - N NVMe Disk (B/s) 200 Time (s) SATA 1000000000
Bandwidth SATA - D 100 500000000 Disk (B/s) 0 0 1 2 3 4 5 6 7 8 9 10 20 30 40 50 1 2 3 4 5 6 7 8 9 1020304050 #VM #VM IT Center DuringDuring the bootstorm,boot storm we we monitored monitored power power absorbed, absorbed, memory memory and CPU and usage. CPU Asusage. expected, As expected, lesser boot lesser time bootcorresponds time Università di Pisa to significant energy savings: boot time using NVMe is 44% less than boot time using SATA when booting 50 VMs, and correspondsP.I.: 00286820501 to significant C.F.: 80003670504 energy savings: boot time using NVMe is 44% less thanLargo boot Bruno time Pontecorvo, using 3, I -SATA56127 Pisa when Italy bootingenergy used 50 VMs, is 41% and less. energy used is 41% less. By using NVMeBy using SSD, NVMe bootstorm SSD bootstorm gets a significant gets a significant speed-up leadingspeedup to leading faster bootto faster and reduced downtime for VMs afterboot a reboot and reduced downtime for VMs after a reboot *Other names and brands may be claimed as the property of others. It can be easily observed that the system uses more resources in terms of memory and CPU when booting VMs using NVMeIt can beinterface: easily observedthis is due thatto less the idle system time spent uses waitingmore resources for data from in terms the drive; of memory in particular and page CPU faultswhen are booting the basic VMs primitiveusing NVMe used interface:by Windows this to isimplement due to less memory idle time mapping, spent and waiting therefore data file from reading. the drive; in particular page faults are the basic primitive used by Windows to implement memory mapping, and therefore file reading.
Energy (mJ) Page fault / sec 150000000 5000 4000 100000000 3000 2000 50000000 1000 0 0 1 2 3 4 5 6 7 8 9 10 20 30 40 50 1 2 3 4 5 6 7 8 9 10 20 30 40 50
NVMe SATA NVMe SATA
CPU 1,80 1 i.e.1,60 when upon certain conditions the systems has to start a large number of virtual machines simultaneously 1,40 NVMe - % CPU - Boot 1,20 NVMe - % Interrupt CPU - Boot 1,00 NVMe - % Privileged CPU - Boot 0,80 SATA - % CPU - Boot 0,60 0,40 SATA - % Interrupt CPU - Boot 0,20 SATA - % Privileged CPU - Boot 0,00 1 2 3 4 5 6 7 8 9 10 20 30 40 50
We can conclude that boot storm benefits from the superior bandwidth offered by NVMe drives, and the advantage is tangible both in boot time and system efficiency leading to better use of the energy necessary for the task.
NVMe interface offers benefits to bootstorm starting from just 5 VMs
Virtual disk test analysis
Once virtual machines finish the boot, diskspd is run to simulate sequential and random access (with 128KB and 4KB block size respectively2) both for read and write. Each test of the four possible combinations has been run for 10 seconds. However, since VMs do not finish boot simultaneously the I/O generated, the four tests were only partly overlapped leading to disk access patterns more realistic than may appear at first sight.
2 These block sizes have been used as representative of many workloads and benchmarks.
*Other names and brands may be claimed as the property of others. IT Center Università di Pisa P.I.: 00286820501 C.F.: 80003670504 Largo Bruno Pontecorvo, 3, I-56127 Pisa Italy
By using NVMe SSD bootstorm gets a significant speedup leading to faster boot and reduced downtime for VMs after a reboot
It can be easily observed that the system uses more resources in terms of memory and CPU when booting VMs using NVMe interface: this is due to less idle time spent waiting data from the drive; in particular page faults are the basic primitive used by Windows to implement memory mapping, and therefore file reading.
Energy (mJ) Page fault / sec 150000000 5000 4000 100000000 3000 2000 50000000 1000 0 0 1 2 3 4 5 6 7 8 9 10 20 30 40 50 1 2 3 4 5 6 7 8 9 10 20 30 40 50
NVMe SATA NVMe SATA
CPU 1,80 1,60 1,40 NVMe - % CPU - Boot 1,20 NVMe - % Interrupt CPU - Boot 1,00 NVMe - % Privileged CPU - Boot 0,80 SATA - % CPU - Boot 0,60 0,40 SATA - % Interrupt CPU - Boot 0,20 SATA - % Privileged CPU - Boot 0,00 1 2 3 4 5 6 7 8 9 10 20 30 40 50
We can concludeWe can that conclude bootstorm that boot benefits storm benefitsfrom the from superior the superior bandwidth bandwidth offered offered by byNVMe NVMe drives, drives, and and the the advantage is tangible bothadvantage in boot istime tangible and system both in bootefficiency, time and leading system toefficiency better leadinguse of the to better energy use necessary of the energy for thenecessary task. for the task. NVMe interface offers benefits to bootstorm starting from just five VMs Virtual DiskNVMe Test interface Analysis offers benefits to bootstorm starting from just 5 VMs IT Center Once virtualVirtual machines finish disk the testboot, diskspd analysis is run to simulate sequential and random access (with 128KB and 4KB Università2 di Pisa block size respectivelyP.I.: 00286820501) both C.F.: 80003670504 for read and write. Each test of the four possible combinationsLargo Bruno Pontecorvo, has 3,been I-56127 run Pisa Italyfor 10 seconds. However, sinceOnce VMs virtual do machines not finish finish boot the simultaneously, boot, diskspd is therun I/Oto simulate generated sequential in the andfour randomtests wereaccess only (with partly 128KB overlapped, and 2 leading Weto disk 4KBmeasured accessblock sizeseveral patterns respectively parameters that are) both duringmore for realisticreadthe test and executionthan write. may Each and appear test compute of atthe first four average sight.possible and standardcombinations deviation. has been For run the forsake 10 of seconds. brevity andHowever, clarity sincewe discuss VMs do the not average finish bootbandwidth simultaneously measured the within I/O generated,the running the virtual four machines, tests were We measured several parameters during the test execution and computed average and standard deviation. For the sake and onlythe powerpartly overlappedconsumption leading as measured to disk accessfrom the patterns operating more system. realistic The than rest may of the appear data atwill first be sight. made available of brevity and clarity we discuss the average bandwidth measured within the running virtual machines, and the power as part of this technical report. consumption as measured from the operating system. The rest of the data will be made available as part of this technicalIn report.the following graph we report the average bandwidth measured for the 8 different tests (four tests for each
kind2 of interface). For both NVMe and SATA interfaces the average bandwidth decreases as the number of In the following These graph block we sizes report have beenthe average used as representative bandwidth measured of many workloads for the eight and benchmarks. different tests (four tests for each kind virtual machines competing for the drive increase. From the graph it is evident how virtual drives residing on of interface). For both NVMe and SATA interfaces the average bandwidth decreases as the number of virtual machines NVMe*Other SSD names drive and always brands mayperform be claimed significantly as the property better of others. as the SATA controller constrain the potential throughput of competing for the drive increases. From the graph it is evident how virtual drives residing on NVMe SSD drive always performthe significantly SSD media. better as the SATA controller constrains the potential throughput of the SSD media.
NVMe SSDs offerNVMe significant SSDs offer significantimprovements improvements in bandwidth in bandwidth and latency and latencyin accessing in virtual drives, key elements in scenariosaccessing such virtual as VDI drives, key elements in scenarios such as VDI
A singleA disk single accessed disk accessed through through the NVMe the NVMe interface interface offers offers almost almost three 3x more times bandwidth more bandwidth for accessing for theaccessing virtual the virtual diskdisk toto eacheach VMVM with with respect respect to toSATA. SATA. This This is very is very important important since since with 50with virtual 50 virtualmachines machines accessing accessing the drive simultaneously, the average bandwidth available for each VM on an NVMe SSD drive (almost 300MB/s) is more than half simultaneously the drive, the average bandwidth available for each VM on an NVMe SSD drive (almost 300MB/s) of the bandwidth of a dedicated SATA SSD drive. This number is significant since nowadays SATA SSD drive performance is more than half of the bandwidth of a dedicated SATA SSD drive. This number is significant since nowadays is often taken as a measure of high performance of a disk drive. SATA SSD drive performance is often taken as a measure of high performance of a disk drive.
Virtual disk bandwidth (MB/s) 3000 NVMe - Rand 4K read avg. MB/s 2500 NVMe - Rand 4K write avg. MB/s 2000 NVMe - Seq 128K read avg. MB/s
1500 NVMe - Seq 128K write avg. MB/s SATA - Rand 4K read avg. MB/s 1000 SATA - Rand 4K write avg. MB/s 500 SATA - Seq 128K read avg. MB/s
0 SATA - Seq 128K write avg. MB/s 1 2 3 4 5 6 7 8 9 10 20 30 40 50
The following tables shows the data used to generate the graph above with the details in measured bandwidth 2 These blockwith sizesdifferent have beenconfigurations. used as being We representative compared theof many standard workloads deviation and benchmarks. of the data set and found the data obtained measuring NVMe disk and SATA are in similar ranges indicating that average comparison is meaningful.
NVMe SATA #VM Rand Rand Seq Seq Rand Rand Seq Seq 4K 4K 128K 128K 4K 4K 128K 128K read write read write read write read write MB/s MB/s MB/s MB/s MB/s MB/s MB/s MB/s 1 2188 2183 2504 2519 511 512 513 515 2 1123 1478 1680 1220 371 399 373 357 3 745 1164 1212 805 253 290 298 229 4 737 1034 951 679 130 222 241 159 5 462 777 715 482 277 235 223 255 6 600 785 672 509 189 165 196 199
*Other names and brands may be claimed as the property of others. IT Center Università di Pisa P.I.: 00286820501 C.F.: 80003670504 Largo Bruno Pontecorvo, 3, I-56127 Pisa Italy
We measured several parameters during the test execution and compute average and standard deviation. For the sake of brevity and clarity we discuss the average bandwidth measured within the running virtual machines, and the power consumption as measured from the operating system. The rest of the data will be made available as part of this technical report.
In the following graph we report the average bandwidth measured for the 8 different tests (four tests for each kind of interface). For both NVMe and SATA interfaces the average bandwidth decreases as the number of virtual machines competing for the drive increase. From the graph it is evident how virtual drives residing on NVMe SSD drive always perform significantly better as the SATA controller constrain the potential throughput of the SSD media.
NVMe SSDs offer significant improvements in bandwidth and latency in accessing virtual drives, key elements in scenarios such as VDI
A single disk accessed through the NVMe interface offers almost 3x more bandwidth for accessing the virtual disk to each VM with respect to SATA. This is very important since with 50 virtual machines accessing simultaneously the drive, the average bandwidth available for each VM on an NVMe SSD drive (almost 300MB/s) is more than half of the bandwidth of a dedicated SATA SSD drive. This number is significant since nowadays SATA SSD drive performance is often taken as a measure of high performance of a disk drive.
Virtual disk bandwidth (MB/s) 3000 NVMe - Rand 4K read avg. MB/s 2500 NVMe - Rand 4K write avg. MB/s 2000 NVMe - Seq 128K read avg. MB/s
1500 NVMe - Seq 128K write avg. MB/s SATA - Rand 4K read avg. MB/s 1000 SATA - Rand 4K write avg. MB/s 500 SATA - Seq 128K read avg. MB/s
0 SATA - Seq 128K write avg. MB/s 1 2 3 4 5 6 7 8 9 10 20 30 40 50
The following table shows the data used to generate the graph above, with the details of measured bandwidth with The following tables shows the data used to generate the graph above with the details in measured bandwidth different configurations. We compared the standard deviation of the data set and found the data obtained measuring NVMewith differentdisk and SATAconfigurations. are in similar We ranges, compared indicating the standard that average deviation comparison of the datais meaningful. set and found the data obtained measuring NVMe disk and SATA are in similar ranges indicating that average comparison is meaningful.
NVMe SATA #VM Rand Rand Seq Seq Rand Rand Seq Seq 4K 4K 128K 128K 4K 4K 128K 128K read write read write read write read write MB/s MB/s MB/s MB/s MB/s MB/s MB/s MB/s 1 2188 2183 2504 2519 511 512 513 515 2 1123 1478 1680 1220 371 399 373 357 3 745 1164 1212 805 253 290 298 229 IT Center 4 737 1034 951 679 130 222 241 159 Università di Pisa P.I.: 00286820501 C.F.: 800036705045 462 777 715 482 277 235 Largo223 Bruno Pontecorvo,255 3, I-56127 Pisa Italy 6 600 785 672 509 189 165 196 199 7 432 690 625 394 172 199 173 150 *Other names and brands may be8 claimed416 as the property565 of others.546 420 148 168 169 163 9 307 490 535 321 133 151 159 148 10 427 575 534 379 128 147 143 134 20 333 419 319 347 147 159 138 138 30 323 433 354 312 117 123 100 110 40 283 324 285 291 100 105 102 102 50 274 280 290 281 91 98 89 95
WeWe measured measured the the power power of ofthe the host host during during the the virtual virtual disk disk test test using using Windows Windows performance performance counters. counters. The Thefollowing graph showsWe measured the difference the power in power of the absorption host during between the virtual the test disk conducted test using usingWindows the NVMeperformance drive and counters. the other The conducted on thefollowing SATA drive. graph shows the difference in power absorption between the test conducted using NVMe drive and the other conducted on the SATA drive. The noticeable aspect is that the system processes more than four times the data at the cost of less than a 2% increase onThe the noticeable power budget. aspect is that the system processes more than four times of data at the cost of less than 2% increase on the power budget.
Power (mW) 275000
270000
265000
NVMe 260000 NVMe SATA 255000
250000
245000 1 2 3 4 5 6 7 8 9 10 20 30 40 50
We found the virtual disk test very interesting, since often disk I/O is often cause for VM slowdown. The WeWe found found the the virtual virtual disk disk test test very very interesting, interesting, since since disk often I/O is disk often I/O the is often cause cause of VM for slowdown. VM slowdown. The bandwidth The available usingbandwidth a single availabledrive is impossible using a single to achieve drive, throughis impossible a 10Gbps to achieve connection through a 10Gbps connection
Disk interface comparison
In this section we briefly compare the disk bandwidth usage recorded using one NVMe drive and one SATA drive. NB:Using Other the names information and brands may collected be claimed by asthe the scripts property during of others. the whole test we have been able to separate the boot storm phase from the virtual disk performance test phase.
In the following graphs we report the ratio between the measured and bandwidth using the NVMe drive and the one using the SATA drive. The three lines show the bandwidth ratio of the whole run (blue line), of the bootstorm test (red line), and of the virtual disk test (green line).
*Other names and brands may be claimed as the property of others. Disk Interface Comparison
In this section we briefly compare the disk bandwidth usage recorded using one NVMe drive and one SATA drive. Using the information collected by the scripts during the whole test we have been able to separate the bootstorm phase from the virtual disk performance test phase.
In the following graphs we report the ratio between the measured bandwidth using the NVMe drive and the one using the SATA drive. ITThe Center three lines show the bandwidth ratio of the whole run (blue line), of the bootstorm test (red line), and of the virtual diskUniversità test (green di Pisa line). P.I.: 00286820501 C.F.: 80003670504 Largo Bruno Pontecorvo, 3, I-56127 Pisa Italy
Bw NVMe / Bw SATA 5
4
3
2
1
0 1 2 3 4 5 6 7 8 9 10 20 30 §§ 50
Disk Disk - Boot Disk - Test
As we can see from the graph the NVMe interface offers always better performance with respect to SATA, and As weduring can see the from whole the test graph we themeasured NVMe interfacean effective always bandwidth offers usagebetter ofperformance more than fourwith respecttimes in to favor SATA, of andthe formerduring . the whole test we measured an effective bandwidth usage of more than four times in favour of the former. During the bootstormDuring phase, the boot when storm booting phase, a single when VM, booting a single a single SATA VM drive, a wassingle able SATA to deliver drive was enough able performance to deliver enough to fulfill the demand.performance However, toeven fulfill booting the demand. just two However, VMs is enough even bootingto exhaust just the 2 VMs SATA is interface enough toand exhaust makes theobvious SATA the interface benefits of usingand the makesNVMe obviousone. the benefits of using the NVMe one. The ability of NVMe to feed CPU with data reduces idle time and contributes to improving the energy efficiencyThe ability of the of system NVMe to feed CPU with data reducing idle time contribute to improve energy efficiency of the system
Test description
The test has been fully automated using two power shell scripts. The first script has been used to replicate the drive of a preconfigured instance of Windows 10 Enterprise edition and create the virtual machines on each drive (drive D has been mapped to a single SATA SSD drive, drive N has been mapped to a single NVMe drive). The second script performed the testing procedure as follows:
- Let S be the sequence @(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50) - For each N in S perform the following steps 1. Record the time of experiment beginning in a file 2. Start the performance counters recording 3. Start N virtual machines on drive D (SATA) 4. Wait for all the VMs generating a .done file to indicate the end of the VM test script completion 5. Stop the performance counter recording 6. Shutdown the virtual machines 7. Repeat steps 3-6 for drive N (NVMe)
After booting the pre-configured Windows 10 Enterprise image performed autologon and started a power shell script performing the virtual disk test using diskspd. This tool is an open source tool developed by Microsoft and used by its engineering team to evaluate storage performance. It is designed to generate a wide variety of disk patterns given a huge number of command line switches. We restrained ourselves to just four tests (read and write) executed for 10 seconds each:
- Seq128K56Q: Sequential R/W, 128KB block size, 56 queues, single thread - Rand4K56Q: Random R/W, 4KB block size, 56 queues, single thread - Seq1M1Q: Sequential R/W, 1MB block size, 1 queue, single thread
*Other names and brands may be claimed as the property of others. Test Description
The test has been fully automated using two power shell The files containing the binary log generated by perfmon scripts. The first script has been used to replicate the drive and by the power shell script running on each VM tested of a preconfigured instance of Windows 10 Enterprise have been analysed with the help of an F# script included edition and create the virtual machines on each drive in the appendix. The script reads all the counters and (drive D has been mapped to a single SATA SSD drive, diskspd output and generates a CSV file with the following drive N has been mapped to a single NVMe drive). The columns (list grouped by category): second script performed the testing procedure as follows: • VMCount: number of VMs tested • Let S be the sequence @(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50) • DriveKind: kind of drive interface tested (SATA or NVMe) • For each N in S perform the following steps 1. Record the time of experiment beginning in a file • Begin: date and time of test 2. Start the performance counters recording 3. Start N virtual machines on drive D (SATA) • End boot: date and time of boot conclusion (i.e. latest 4. Wait for all the VMs generating a .done file to indicate date and time of disk test start) the end of the VM test script completion 5. Stop the performance counter recording • End test: date and time of test conclusion (i.e. latest 6. Shutdown the virtual machines date and time of disk test end) 7. Repeat steps 3-6 for drive N (NVMe) • Boot duration: boot duration in seconds After booting, the pre-configured Windows 10 Enterprise image performed autologon and started a power shell • Average CPU: all, boot, test script performing the virtual disk test using diskspd. -- CPU Idle % This tool is an open source tool developed by Microsoft -- CPU Interrupt % and used by its engineering team to evaluate storage -- CPU User % performance. It is designed to generate a wide variety -- CPU % of disk patterns given a huge number of command line -- CPU Privileged % switches. We restrained ourselves to just four tests (read and write) executed for 10 seconds each: • Average Disk (C, D – SATA, N – NVMe): all, boot, test -- Read (B/s) • Seq128K56Q: Sequential R/W, 128KB block size, 56 -- Queue Length queues, single thread -- Disk (B/s) • Rand4K56Q: Random R/W, 4KB block size, 56 queues, -- Disk Write (B/s) single thread -- Disk Transfers /sec • Seq1M1Q: Sequential R/W, 1MB block size, 1 queue, single thread • Average Page faults /sec: all, boot, test • Rand4K1Q: Random R/W, 4KB block size, 1 queue, single thread • Average Available memory (MB): all, boot, test
The number of queues has been empirically determined by • Average Power: all, boot, test testing the bandwidth improvement for the Seq128K on a particular volume with different queue numbers; on the • Rand4KR, Rand4KW, Seq128KR, Seq128KW server we tested, 56 seemed to be a good value. -- Bytes (average and variance): bytes transferred -- IOs (average and variance): number of I/O operations The data recorded by the diskspd instances executed in -- MB/s (average and variance): transfer bandwidth each virtual machine, together with the files to record the -- IOps (average and variance): I/O operations per beginning and the end of the execution are saved on a second network share provided by the hypervisor host through an internal network virtual switch. A DHCP server ensures The performance counters data have been analysed for that all the virtual machines receive an IP address in this the whole test and for the boot and test phases, in all internal network. cases the average value has been computed. The data relative to diskspd execution on different VMs have been reported as average and variance.
NB: Other names and brands may be claimed as the property of others. Conclusion Bibliography
In this report we discussed the results of the experiments 1. http://www.nvmexpress.org/ on the impact of PCIe NVMe SSD disk drives with respect to SATA SSD disk drives, in the bootstorm of virtual 2. http://ark.intel.com/products/82934/Intel-SSD-DC- machines and virtual disk benchmarking scenarios. In S3610-Series-1_6TB-2_5in-SATA-6Gbs-20nm-MLC both scenarios NVMe disk drives have shown significant improvements, capable of affecting real-world workloads. 3. http://ark.intel.com/products/80992/Intel-SSD-DC- P3600-Series-1_6TB-12-Height-PCIe-3_0-20nm-MLC The bootstorm test showed how the interface led to substantial speed-up of the simultaneous boot of many 4. http://ark.intel.com/products/81055/Intel-Xeon- virtual machines starting from just five concurrent VMs. Processor-E5-2683-v3-35M-Cache-2_00-GHz
When we performed diskspd benchmarking from within 5. http://www.itc.unipi.it/index.php/2016/02/23/ running virtual machines, we found that the NVMe comparison-of-solid-state-drives-ssds-on-different- interface is capable of delivering significant bandwidth to bus-interfaces/ each VM. In particular, with 50 VMs running on a single hypervisor the average virtual disk bandwidth recorded is 6. http://www.itc.unipi.it/index.php/2016/05/01/solid- of almost 300MB/s, equivalent to assigning to each VM state-software-defined-storage-NVMe-vs-sata/ a dedicated disk performing at more than half of a SATA SSD drive. Again, benefits from using NVMe drives are 7. https://github.com/Microsoft/diskspd apparent even when running a small number of virtual machines. 8. https://downloadcenter.intel.com/it/ download/25771/-Driver-RAID-Intel-Rapid-Storage- Results show that using NVMe local drives provides Technology-Enterprise-NVMe-Intel-RSTe-NVMe- significant increment of disk bandwidth in the context of virtualization, allowing better use of available cores and 9. http://social.technet.microsoft.com/wiki/contents/ memory on single hypervisors. The available bandwidth articles/15198.storage-spaces-overview.aspx offered by NVMe drives is greater than the 10Gbps network connection that is often used to access remote storage, contributing significantly to improving the density and efficiency of virtualization systems. Numbers, also from previous reports, indicate that this will contribute to pushing hyperconverged systems with software defined storage to better exploit the improved capability of SSDs and PCIe interface using the NVMe protocol. Appendix Hardware Configuration
Tests have been performed on a Dell* R630 with the following configuration:
• Dell R630 with support for up to four NVMe drives
• Perc* H730 Mini controller for Boot drive
• 2 Intel® Xeon® Processor E5-2683 v3 2GHz CPU
• 2 Intel® SSD DC S3710 Series (SATA boot drive)
• 4 Intel® SSD DC P3600 Series (NVMe PCIe)
• 4 Intel® SSD DC S3610 Series (SATA)
• 128Gb (8x16Gb) DDR4R4 RAM System Setup
The system has been installed with Windows Server 2012 R2 Standard edition with the drivers provided by Dell for the R630 and the Intel® Solid State Drive Data Center Family for NVMe Drivers. We then installed all the latest updates from Microsoft. The SATA RAID hardware control for Intel SSD DC S3610 Series has been configured in pass-through mode. The system has joined the AD of our network. The versions of software used for testing are diskspd 2.0.15, and Intel RSTe 4.5.0.2072.
Test Scripts
For completeness we include the full source of the scripts used for our test. VM Creation Script
$basedir = “C:\Users\cisterni\desktop”
$nvme0 = Get-PhysicalDisk -FriendlyName PhysicalDisk5 $nvme1 = Get-PhysicalDisk -FriendlyName PhysicalDisk6 $nvme2 = Get-PhysicalDisk -FriendlyName PhysicalDisk7 $nvme3 = Get-PhysicalDisk -FriendlyName PhysicalDisk8
$sata0 = Get-PhysicalDisk -FriendlyName PhysicalDisk0 $sata1 = Get-PhysicalDisk -FriendlyName PhysicalDisk1 $sata2 = Get-PhysicalDisk -FriendlyName PhysicalDisk2 $sata3 = Get-PhysicalDisk -FriendlyName PhysicalDisk3 function createVolume ([string] $poolName, [string] $resiliency, [char] $letter, [string] $filesystem=”NTFS”) { $name = $poolName + “_vd” $drive = New-VirtualDisk -StoragePoolFriendlyName $poolName -ResiliencySettingName $resiliency -ProvisioningType Fixed -UseMaximumSize -FriendlyName $name initialize-disk -VirtualDisk $drive $vol = New-Partition -DiskId $drive.UniqueId -UseMaximumSize -DriveLetter $letter $vol | Format-Volume -Confirm:$false -FileSystem $filesystem -NewFileSystemLabel $poolName return $drive } $nvmes = @($nvme0, $nvme1, $nvme2, $nvme3) $satas = @($sata0, $sata1, $sata2, $sata3) $nvmepool = New-StoragePool -FriendlyName NVMe-VM -PhysicalDisks $nvmes - StorageSubSystemFriendlyName “Storage Spaces on intelssd” $satapool = New-StoragePool -FriendlyName SATA-VM -PhysicalDisks $satas - StorageSubSystemFriendlyName “Storage Spaces on intelssd” createVolume -poolName NVMe-VM -resiliency Simple -letter N createVolume -poolName SATA-VM -resiliency Simple -letter D mkdir N:\VM mkdir D:\VM function CreateVM([int] $number, [char] $letter) { echo “Creating vdi-$number...” $vmname = “$letter-vdi-$number” $vmdest = $letter + “:\VM” $diskdest = $letter + “:\VM\$vmname.vhdx” copy $basedir\Win10EntGold.vhdx $diskdest $vm = new-vm -Name $vmname -path $vmdest -Generation 2 $vm | Add-VMHardDiskDrive -Path $diskdest $vm.NetworkAdapters[0] | Remove-VMNetworkAdapter $vm | Add-VMNetworkAdapter -SwitchName InternalNet $vm | Set-VMMemory -StartupBytes 536870912 -MinimumBytes 536870912 -MaximumBytes 2147483648 -DynamicMemoryEnabled $true $vm | Set-VMProcessor -Count 2 } function CreateVMs([int] $num, [char] $letter, [int] $from = 0) { for ($i = $from; $i -lt $num; $i++) { CreateVM -number $i -letter $letter } }
CreateVMs -num 50 -letter N CreateVMs -num 50 -letter D VM Coordination Script
$collectors = New-Object -COM Pla.DataCollectorSetCollection; $sharedir = “C:\Users\cisterni\Desktop\BootStormLogs\Coord”; $resdir = “C:\Users\cisterni\Desktop\BootstormExp”; function GetBootstormCollector() { $collectors.GetDataCollectorSets($null, “Service\*”); $bootstormCollector = $null; $collectors._NewEnum | % { if ($_.Name -eq “Bootstorm”) { $bootstormCollector = $_; echo $_ } }; } function StartVMs ([int] $num, [char] $letter) { $vms = @() for ($i = 0; $i -lt $num; $i++) { $vms += Get-VM “$letter-vdi-$i” } start-vm -VM $vms } function StopVMs ([int] $num, [char] $letter) { $vms = @() for ($i = 0; $i -lt $num; $i++) { $vms += Get-VM “$letter-vdi-$i” } Stop-VM -VM $vms } function WaitWorkloads ([string] $basedir = $sharedir, [string] $filepattern = “*.done”, [int] $count) { Do { sleep -s 1 $m = Get-ChildItem “$basedir\$filepattern” | measure Write-Progress -Activity “Wait for tests” -Status (“Waiting for “ + ($count - $m.Count)) -PercentComplete ($m.Count / $count) } While ( $m.Count -lt $count )
Write-Progress -Activity “Wait for tests” -Completed -Status “Tests completed” } function makeTest ([int] $count, [char] $letter) { Write-Progress -Activity “Test $count VMs on $letter” $bootstormCollector = GetBootstormCollector
mkdir (“$resdir\exp-VM$count-Drv$letter”)
Get-Date -Format “yyyy-MM-dd HH:mm:ss” > “$resdir\exp-VM$count-Drv$letter\exp.begin”
$bootstormCollector.start($true)
StartVMs -num $count -letter $letter
WaitWorkloads -count $count
$bootstormCollector.Stop($true)
StopVMs -num $count -letter $letter Write-Progress -Activity “Test $count VMs on $letter” -Status “Moving results” # refresh the data associated with the collector $bootstormCollector = GetBootstormCollector $loc = $bootstormCollector.LatestOutputLocation
icacls $loc /grant cisterni:F
mv -Force (“$sharedir\*”) $loc
mv -Force $loc (“$resdir\exp-VM$count-Drv$letter”)
Write-Progress -Activity “Test $count VMs on $letter” -Completed -Status “Done.” } function makeTestRound ([int] $count) { makeTest -count $count -letter ‘D’ makeTest -count $count -letter ‘N’
} function testSuite() { @(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50) | ForEach-Object { makeTestRound -count $_ } } testSuite
VM Diskspd Test Script
$dest = “\\192.168.10.2\Coord” function GetIP([string] $base) { $ip = $null Get-NetIPAddress | where { $_.IPv4Address -match $base } | ForEach-Object { $ip = $_.IPv4Address } $ip }
$ip = GetIP -base “192.168.10.”
Get-Date -Format “yyyy-MM-dd HH:mm:ss” > “$dest\$ip.start”
C:\users\admin\Desktop\Diskspd-v2.0.15\amd64fre\diskspd.exe -b128K -d10 -o56 -t1 -W -h -w0 C:\Users\admin\Desktop\testfile.dat > “$dest\$ip-seq128Kr.txt” C:\users\admin\Desktop\Diskspd-v2.0.15\amd64fre\diskspd.exe -b128K -d10 -o56 -t1 -W -h -w0 C:\Users\admin\Desktop\testfile.dat > “$dest\$ip-seq128Kw.txt” C:\users\admin\Desktop\Diskspd-v2.0.15\amd64fre\diskspd.exe -b128K -d10 -o56 -t1 -r -W -h -w0 C:\Users\admin\Desktop\ testfile.dat > “$dest\$ip-rand4Kr.txt” C:\users\admin\Desktop\Diskspd-v2.0.15\amd64fre\diskspd.exe -b128K -d10 -o56 -t1 -r -W -h -w0 C:\Users\admin\Desktop\ testfile.dat > “$dest\$ip-rand4Kw.txt”
Get-Date -Format “yyyy-MM-dd HH:mm:ss” > “$dest\$ip.done” F# Script For Parsing Performance Counters .blg Files And Diskspd Outputs
#I __SOURCE_DIRECTORY__ #r @”FSharp.Management.0.4.1\lib\net40\System.Management.Automation.dll” #r @”FSharp.Management.0.4.1\lib\net40\FSharp.Management.PowerShell.dll” #r @”C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Microsoft.PowerShell.Commands.Diagnostics\ v4.0_3.0.0.0__31bf3856ad364e35\Microsoft.PowerShell.Commands.Diagnostics.dll” open FSharp.Management open System.IO open System.Text.RegularExpressions open System.Collections.Generic
// Counter processing type CounterLabels = Idle | Interrupt | User | Processor | Privileged | DiskReadBps | CurrentDiskQueueLength | DiskBps | DiskWriteBps | DiskTransfers | PageFaultsSec | AvailableMB | Power let counterColumns = dict [ (Idle, 0) (Interrupt, 1) (User, 2) (Processor, 3) (Privileged, 4) (DiskReadBps, 0) (CurrentDiskQueueLength, 1) (DiskBps, 2) (DiskWriteBps, 3) (DiskTransfers, 4) (PageFaultsSec, 0) (AvailableMB, 1) (Power, 0) ] let counterColumnsLabels = dict
[ (Idle, “CPU Idle %”) (Interrupt, “CPU Interrupt %”) (User, “CPU User %”) (Processor, “CPU %”) (Privileged, “CPU Privileged %”) (DiskReadBps, “Disk Read (B/s)”) (CurrentDiskQueueLength, “Queue Length”) (DiskBps, “Disk (B/s)”) (DiskWriteBps, “Disk Write (B/s)”) (DiskTransfers, “Disk Transfers /sec”) (PageFaultsSec, “Page fauls /sec”) (AvailableMB, “Available memory (MB)”) (Power, “Power”) ] let getCounterNames (set:Microsoft.PowerShell.Commands.GetCounter.PerformanceCounterSampleSet) = set.CounterSamples |> Seq.map (fun v -> v.Path) let getColumn (set:seq
for x in data do n <- n + 1 let delta = x - mean mean <- mean + delta/float(n) M2 <- M2 + delta*(x - mean)
if n < 2 then None else Some(mean, M2 / float(n - 1)) type PS = PowerShellProvider< “Microsoft.PowerShell.Management;Microsoft.PowerShell.Core” > let ImportCounter (fn:string) = match PS.``Import-Counter``([| fn |], summary=false) with | Success v -> v.ImmediateBaseObject :?> System.Collections.ObjectModel.Collection
[
\r\navg\.\|.*?([\d\.]+)%\|.*?([\d\.]+)%\|.*?([\d\.]+)%\|.*?([\d\.]+)%”) let pP (gn:int) = (cpud.Groups.[gn].Value |> float) * 1.; BlockSize : float; AccessType : DiskAccessType; Queues : int; EffectiveDuration : float; ThreadCount : int; CPUCount : int } let matchMeta teststring = let testd = Regex.Match(teststring, “duration:.*?(\d+)s.*?block size:.*?(\d+).*?using (random|sequential) I/O.*?number of outstanding I/O operations:.*?(\d+).*?actual test time:.*?([\d\.]+)s.*?thread count:.*?(\d+).*?proc count:.*?(\d+)”, RegexOptions.Singleline) let pS (gn:int) = (testd.Groups.[gn].Value |> float) * 1. let pB (gn:int) = (testd.Groups.[gn].Value |> float) * 1. let pI (gn:int) = (testd.Groups.[gn].Value |> int) let dT (gn:int) = if testd.Groups.[gn].Value = “random” then Random else Sequential { Duration = pS 1; BlockSize = pB 2; AccessType = dT 3; Queues = pI 4; EffectiveDuration = pS 5; ThreadCount = pI 6; CPUCount = pI 7 } type DiskSpdThreadPerf = { Mode : DiskAccessMode; BytesCount : int64; IOs : int64
{ Mode = Read; BytesCount = pB iosr 1; IOs = pI iosr 2; MBps = pM iosr 3; IOps = pIs iosr 4 }, { Mode = Write; BytesCount = pB iosw 1; IOs = pI iosw 2; MBps = pM iosw 3; IOps = pIs iosw 4 } ) type DiskSpdExp = { BlockSize : float
member this.Name = name member this.Begin with get() = vmbegin and set (v) = vmbegin <- v member this.End with get() = vmend and set (v) = vmend <- v member this.Experiments = experiments let cpucols = [ Idle; Interrupt; User; Processor; Privileged ] let diskcols = [DiskReadBps; CurrentDiskQueueLength; DiskBps; DiskWriteBps; DiskTransfers ] let memcols = [ PageFaultsSec; AvailableMB ] let powcols = [ Power ] let processExp (d, out:StreamWriter) = printfn “Processing %s” d let expm = Regex.Match(d, @”\\exp\-VM(\d+)\-Drv([DN])\\”) let vmcount = expm.Groups.[1].Value |> int let drive = if expm.Groups.[2].Value = “D” then SATA else NVMe let begexp = File.ReadAllLines(d + @”\..\exp.begin”).[0] |> System.DateTime.Parse let data = new System.Collections.Generic.Dictionary
let name = Regex.Match(fi.Name, “(.*)\\.done”).Groups.[1].Value let endt = File.ReadAllLines(f).[0] |> System.DateTime.Parse if not(data.ContainsKey(name)) then data.Add(name, new VMExp(name)) data.[name].End <- endt | “.start” -> let name = Regex.Match(fi.Name, “(.*)\\.start”).Groups.[1].Value let start = File.ReadAllLines(f).[0] |> System.DateTime.Parse if not(data.ContainsKey(name)) then data.Add(name, new VMExp(name)) data.[name].Begin <- start | _ -> () ) let endboot = data |> Seq.map (fun v -> v.Value.Begin ) |> Seq.max let endtest = data |> Seq.map (fun v -> v.Value.End) |> Seq.max let splitCounters name = let c = counters.[name] let cb = c |> Seq.filter (fun v -> v.Timestamp <= endboot) let ct = c |> Seq.filter (fun v -> v.Timestamp > endboot) (c, cb, ct) let avgColumn counters col = let c, cb, ct = counters let ac = counterColumns.[col] |> getTimestampedColumn c |> avgTimestampedColumn let acb = counterColumns.[col] |> getTimestampedColumn cb |> avgTimestampedColumn let act = counterColumns.[col] |> getTimestampedColumn ct |> avgTimestampedColumn (ac, acb, act)
let pdate (d:System.DateTime) = d.ToString(“yyyy-MM-dd HH:mm:ss”) let chgcomma (s:string) = s.Replace(“.”, “,”) let pvs (l:CounterLabels list) (d:IDictionary
out.Write(sprintf “%d;%A;%s;%s;%s;%s;” vmcount drive (pdate begexp) (pdate endboot) (pdate endtest) ((sprintf “%A” (endboot - begexp).TotalSeconds) |> chgcomma))
let cpu = splitCounters “CPU” let cpuvals = cpucols |> Seq.map (fun v -> (v, avgColumn cpu v)) |> Seq.toList |> dict
let diskC = splitCounters “DiskC” let diskCvals = diskcols |> Seq.map (fun v -> (v, avgColumn diskC v)) |> Seq.toList |> dict
let diskD = splitCounters “DiskD_SATA” let diskDvals = diskcols |> Seq.map (fun v -> (v, avgColumn diskD v)) |> Seq.toList |> dict
let diskN = splitCounters “DiskN_NVMe” let diskNvals = diskcols |> Seq.map (fun v -> (v, avgColumn diskN v)) |> Seq.toList |> dict let mem = splitCounters “Memory”
let memvals = memcols |> Seq.map (fun v -> (v, avgColumn mem v)) |> Seq.toList |> dict
let power = splitCounters “Power” let powervals = powcols |> Seq.map (fun v -> (v, avgColumn power v)) |> Seq.toList |> dict [0; 1; 2] |> Seq.iter (pvs cpucols cpuvals) [0; 1; 2] |> Seq.iter (pvs diskcols diskCvals) [0; 1; 2] |> Seq.iter (pvs diskcols diskDvals) [0; 1; 2] |> Seq.iter (pvs diskcols diskNvals) [0; 1; 2] |> Seq.iter (pvs memcols memvals) [0; 1; 2] |> Seq.iter (pvs powcols powervals)
let avg (l:seq
let outExpStat accessType accessMode = let exp = data |> Seq.map (fun vm -> vm.Value.Experiments |> Seq.filter (fun exp -> exp.AccessType = accessType && exp.AccessMode = accessMode) |> Seq.head) let expb, expb_var = exp |> Seq.map (fun exp -> let p, _, _ = exp.Throughput in p.BytesCount |> float) |> avg let expio, expio_var = exp |> Seq.map (fun exp -> let p, _, _ = exp.Throughput in p.IOs |> float) |> avg let expmbps, expmbps_var = exp |> Seq.map (fun exp -> let p, _, _ = exp.Throughput in p.MBps |> float) |> avg let expiops, expiops_var = exp |> Seq.map (fun exp -> let p, _, _ = exp.Throughput in p.IOps |> float) |> avg out.Write((sprintf “%f;%f;%f;%f;%f;%f;%f;%f;” expb expb_var expio expio_var expmbps expmbps_var expiops expiops_var).Replace(“.”, “,”))
outExpStat DiskAccessType.Random DiskAccessMode.Read outExpStat DiskAccessType.Random DiskAccessMode.Write outExpStat DiskAccessType.Sequential DiskAccessMode.Read outExpStat DiskAccessType.Sequential DiskAccessMode.Write
out.WriteLine() let processExpDir (out:StreamWriter) = let printHeads cols pfx sfx = out.Write(sprintf “%s;” (System.String.Join(“;”, cols |> Seq.map (fun v -> pfx + counterColumnsLabels.[v] + sfx) |> Seq.toArray)))
out.Write(“VMCount;DriveKind;Begin;End boot;End test;Boot duration;”) [ (cpucols, “”, “”); (cpucols, “”, “ - Boot”); (cpucols, “”, “ - Test”) (diskcols, “DiskC”, “”); (diskcols, “DiskC”, “ - Boot”); (diskcols, “DiskC”, “ - Test”) (diskcols, “DiskD (SATA)”, “”); (diskcols, “DiskD (SATA)”, “ - Boot”); (diskcols, “DiskD (SATA)”, “ - Test”) (diskcols, “DiskN (NVMe)”, “”); (diskcols, “DiskN (NVMe)”, “ - Boot”); (diskcols, “DiskN (NVMe)”, “ - Test”) (memcols, “”, “”); (memcols, “”, “ - Boot”); (memcols, “”, “ - Test”) (powcols, “”, “”); (powcols, “”, “ - Boot”); (powcols, “”, “ - Test”) ] |> Seq.iter (fun (c, p, s) -> printHeads c p s)
out.Write(“Rand4KR Avg. Bytes;Rand4KR Var. Bytes;Rand4KR Avg. IOs;Rand4KR Var. IOs;Rand4KR Avg. MB/s;Rand4KR Var. MB/s;Rand4KR Avg. IOps;Rand4KR Var. IOps;”)
out.Write(“Rand4KW Avg. Bytes;Rand4KW Var. Bytes;Rand4KW Avg. IOs;Rand4KW Var. IOs;Rand4KW Avg. MB/s;Rand4KW Var. MB/s;Rand4KW Avg. IOps;Rand4KW Var. IOps;”) out.Write(“Seq128KR Avg. Bytes;Seq128KR Var. Bytes;Seq128KR Avg. IOs;Seq128KR Var. IOs;Seq128KR Avg. MB/s;Seq128KR Var. MB/s;Seq128KR Avg. IOps;Seq128KR Var. IOps;”) out.Write(“Seq128KW Avg. Bytes;Seq128KW Var. Bytes;Seq128KW Avg. IOs;Seq128KW Var. IOs;Seq128KW Avg. MB/s;Seq128KW Var. MB/s;Seq128KW Avg. IOps;Seq128KW Var. IOps”) out.WriteLine();
Directory.EnumerateDirectories(__SOURCE_DIRECTORY__) |> Seq.filter (fun d -> Regex.IsMatch(d, @”exp\-VM\d+\-Drv[DN]”)) |> Seq.map (fun d -> Directory.GetDirectories(d) |> Seq.head) |> Seq.iter (fun d -> processExp(d, out)) let outn = __SOURCE_DIRECTORY__ + @”\summary.csv” if File.Exists(outn) then File.Delete(outn) let out = File.CreateText(outn) processExpDir(out) out.Close() out.Dispose() Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com.
Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel. com/performance.
Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate.
Intel, the Intel logo, Intel® Xeon®, Intel® SSD DC S3610, Intel® SSD DC S3710, and Intel® SSD DC P3600 are trademarks of Intel Corporation in the U.S. and/or other countries. © 2017 Dell EMC, All rights reserved.
Dell EMC, the Dell EMC logo and products — as identified in this document — are registered trademarks of Dell, Inc. in the U.S.A. and/or other countries. No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose without the written permission of Dell, Inc. (“Dell”).
Dell EMC disclaims proprietary interest in the marks and names of others. Dell EMC service offerings do not affect customer’s statutory rights.
Availability and terms of Dell EMC Services vary by region. Terms and Conditions of Sales, Service and Finance apply and are available on request or at Dell.co.uk/terms.
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.
Dell Corporation Limited. Registered in England. Reg. No. 02081369 Dell House, The Boulevard, Cain Road, Bracknell, Berkshire, RG12 1LF, UK.