CS147 CS 147: Computer Systems Performance Analysis Workload Selection 2015-06-15 CS 147: Computer Systems Performance Analysis Workload Selection 1 / 39 Overview CS147 Overview Workload Types What is a Workload? Instruction Workloads Synthetic Workloads Standard Benchmarks Exercisers and Drivers Workload Selection Overview Considerations Example 2015-06-15 Workload Types What is a Workload? Instruction Workloads Synthetic Workloads Standard Benchmarks Exercisers and Drivers Workload Selection Considerations Example 2 / 39 Workload Types What is a Workload? What is a Workload? CS147 What is a Workload? Workload Types I Workload: anything a computer is asked to do I Test workload: any workload used to analyze performance What is a Workload? I Real workload: any observed during normal operations Synthetic workload: created for controlled testing What is a Workload? I 2015-06-15 I Workload: anything a computer is asked to do I Test workload: any workload used to analyze performance I Real workload: any observed during normal operations I Synthetic workload: created for controlled testing 3 / 39 Workload Types What is a Workload? Real Workloads CS147 Real Workloads Workload Types I Advantage: represent reality I Disadvantage: uncontrolled I Can’t be repeated What is a Workload? I Can’t be described simply I Difficult to analyze I Nevertheless, often useful for “final analysis” papers Real Workloads I E.g., “We ran system foo and it works well” 2015-06-15 I Advantage: represent reality I Disadvantage: uncontrolled I Can’t be repeated I Can’t be described simply I Difficult to analyze I Nevertheless, often useful for “final analysis” papers I E.g., “We ran system foo and it works well” 4 / 39 Workload Types What is a Workload? Synthetic Workloads CS147 Synthetic Workloads Workload Types I Advantages: I Controllable I Repeatable What is a Workload? I Portable to other systems I Easily modified Synthetic Workloads I Disadvantage: can never be sure real world will be the same 2015-06-15 I Advantages: I Controllable I Repeatable I Portable to other systems I Easily modified I Disadvantage: can never be sure real world will be the same 5 / 39 Workload Types Instruction Workloads Instruction Workloads CS147 Instruction Workloads Workload Types I Useful only for CPU performance I But teach useful lessons for other situations I Development over decades I “Typical” instruction (ADD) Instruction Workloads I Instruction mix (by frequency of use) I Sensitive to compiler, application, architecture I Still used today (GFLOPS) I Processor clock rate Instruction Workloads I Only valid within processor family 2015-06-15 I Useful only for CPU performance I But teach useful lessons for other situations I Development over decades I “Typical” instruction (ADD) I Instruction mix (by frequency of use) I Sensitive to compiler, application, architecture I Still used today (GFLOPS) I Processor clock rate I Only valid within processor family 6 / 39 Workload Types Instruction Workloads Instruction Workloads (cont’d) CS147 Instruction Workloads (cont’d) Workload Types I Modern complexity makes mixes invalid I Pipelining I Data/instruction caching I Prefetching Instruction Workloads I Kernel is inner loop that does useful work: I Sieve, matrix inversion, sort, etc. I Ignores setup, I/O, so can be timed by analysis if desired (at Instruction Workloads (cont’d) least in theory) 2015-06-15 I Modern complexity makes mixes invalid I Pipelining I Data/instruction caching I Prefetching I Kernel is inner loop that does useful work: I Sieve, matrix inversion, sort, etc. I Ignores setup, I/O, so can be timed by analysis if desired (at least in theory) 7 / 39 Workload Types Synthetic Workloads Synthetic Workloads CS147 Synthetic Workloads Workload Types I Complete programs I Designed specifically for measurement I May do real or “fake” work Synthetic Workloads I May be adjustable (parameterized) I Two major classes: I Real-world benchmarks Synthetic Workloads I Purpose-written exercisers 2015-06-15 Concern is that real-world benchmarks represent only a specific problem. I Complete programs Concern is that exercisers may not stress system the same way as I Designed specifically for measurement real programs (e.g., page faults). I May do real or “fake” work I May be adjustable (parameterized) I Two major classes: I Real-world benchmarks I Purpose-written exercisers 8 / 39 Workload Types Synthetic Workloads Real-World Benchmarks CS147 Real-World Benchmarks Workload Types I Pick a representative application I Pick sample data I Run it on system to be tested Synthetic Workloads I Modified Andrew Benchmark, MAB, is a real-world benchmark I Easy to do, accurate for that sample data Real-World Benchmarks I Fails to consider other applications, data 2015-06-15 I Pick a representative application I Pick sample data I Run it on system to be tested I Modified Andrew Benchmark, MAB, is a real-world benchmark I Easy to do, accurate for that sample data I Fails to consider other applications, data 9 / 39 Workload Types Synthetic Workloads Application Benchmarks CS147 Application Benchmarks Workload Types I Variation on real-world benchmarks I Choose most important subset of functions I Write benchmark to test those functions Synthetic Workloads I Tests what computer will be used for I Need to be sure important characteristics aren’t missed Application Benchmarks I Mix of functions must reflect reality 2015-06-15 I Variation on real-world benchmarks I Choose most important subset of functions I Write benchmark to test those functions I Tests what computer will be used for I Need to be sure important characteristics aren’t missed I Mix of functions must reflect reality 10 / 39 Workload Types Standard Benchmarks CS147 “Standard” Benchmarks “Standard” Benchmarks I Often need to compare general-purpose computer systems for general-purpose use Workload Types I E.g., should I buy an AMD or Intel CPU? I Tougher: Mac or PC? I Desire for an easy, comprehensive answer Standard Benchmarks I People writing articles may need to compare tens of machines I Often need to make comparisons over time I Is this year’s PowerPC faster than last year’s Pentium? I Probably yes, but by how much? “Standard” Benchmarks I Don’t want to spend time writing own code I Could be buggy or not representative I Need to compare against other people’s results 2015-06-15 I Often need to compare general-purpose computer systems I “Standard” benchmarks offer solution for general-purpose use I E.g., should I buy an AMD or Intel CPU? I Tougher: Mac or PC? I Desire for an easy, comprehensive answer I People writing articles may need to compare tens of machines I Often need to make comparisons over time I Is this year’s PowerPC faster than last year’s Pentium? I Probably yes, but by how much? I Don’t want to spend time writing own code I Could be buggy or not representative I Need to compare against other people’s results I “Standard” benchmarks offer solution 11 / 39 Workload Types Standard Benchmarks CS147 Popular “Standard” Benchmarks Popular “Standard” Benchmarks I Sieve, 8 queens, etc. Workload Types I Whetstone I Linpack I Dhrystone I Debit/credit Standard Benchmarks I TPC I SPEC Popular “Standard” Benchmarks I MAB I Winstone, webstone, etc. I Postmark, IOzone, FileBench 2015-06-15 I Sieve, 8 queens, etc. I ... I Whetstone I Linpack I Dhrystone I Debit/credit I TPC I SPEC I MAB I Winstone, webstone, etc. I Postmark, IOzone, FileBench I ... 12 / 39 Workload Types Standard Benchmarks Sieve, etc. CS147 Sieve, etc. Workload Types I Prime number sieve (Erastothenes) I Nested for loops I Often such small array that it’s silly Standard Benchmarks I 8 queens I Recursive I Many others Sieve, etc. I Generally not representative of real problems 2015-06-15 I Prime number sieve (Erastothenes) I Nested for loops I Often such small array that it’s silly I 8 queens I Recursive I Many others I Generally not representative of real problems 13 / 39 Workload Types Standard Benchmarks Whetstone CS147 Whetstone Workload Types I Dates way back (can compare against 70’s) I Based on real observed instruction frequencies I Entirely synthetic (no useful result) Standard Benchmarks I Modern optimizers may delete code I Mixed data types, but best for floating-point Whetstone I Be careful of incomparable variants! 2015-06-15 I Dates way back (can compare against 70’s) I Based on real observed instruction frequencies I Entirely synthetic (no useful result) I Modern optimizers may delete code I Mixed data types, but best for floating-point I Be careful of incomparable variants! 14 / 39 Workload Types Standard Benchmarks LINPACK CS147 LINPACK Workload Types I Based on real programs and data Standard Benchmarks I Developed by supercomputer users I Great if you’re doing serious numerical computation LINPACK 2015-06-15 I Based on real programs and data I Developed by supercomputer users I Great if you’re doing serious numerical computation 15 / 39 Workload Types Standard Benchmarks Dhrystone CS147 Dhrystone Workload Types I Bad pun on “Whetstone” I Motivated by Whetstone’s perceived excessive emphasis on floating point Standard Benchmarks I Dates to when µp’s were integer-only I Still somewhat popular in PC world Dhrystone I Again, watch out for version mismatches 2015-06-15 I Bad pun on “Whetstone” I Motivated by Whetstone’s perceived excessive emphasis on floating point I Dates to when µp’s were integer-only I Still somewhat popular in PC world I Again, watch out for version mismatches 16 / 39 Workload Types Standard Benchmarks Debit/Credit Benchmark CS147 Debit/Credit