Sysmark® 2002 Performance Benchmark: Analysis Presentation Benchmarks & Athlon XP Performance
Total Page:16
File Type:pdf, Size:1020Kb
SYSmark® 2002 Performance Benchmark: Analysis Presentation Benchmarks & Athlon XP Performance • Benchmarks are currently the only way for business and home consumers to clearly measure the real-world performance of PCs. Benchmarks must accurately and objectively represent real-world application performance relevant to the end-user. •BAPCo® is a non-profit consortium, originally founded in 1995 by Intel, IBM, Dell and NCR. SYSmark® is BAPCo’s flagship application benchmark. • The AMD Athlon™ XP processor routinely outperformed comparable Intel Pentium® 4 processors using the SYSmark 2001 benchmark. • On January 31, 2002, BAPCo released SYSmark 2002. 2 SYSmark 2001 & 2002 Similarities Internet Content SYSmark Both SYSmark 2001 & SYSmark Creation 2002 Version 2002 are application-based Adobe Photoshop v6.0.1 benchmarks that attempt to Adobe Premier Same reflect usage patterns for end Macromedia Dreamweaver Same users in the areas of Internet Macromedia Flash Same Content Creation and Office Windows Media Encoder v7.01 Productivity. Office Productivity SYSmark 2002 Version The chart on the right explains Microsoft Word v2002 that SYSmark 2002 uses the same Microsoft Excel v2002 applications as SYSmark 2001 Microsoft Power Point v2002 with updated versions where Microsoft Outlook v2002 applicable. Microsoft Access v2002 Dragon Naturally Speaking Same McAfee VirusScan Same Netscape Communicator Same WinZip Computing Same 3 SYSmark 2002 Inconsistencies AMD tested SYSmark 2001 and 2002 on similarly configured systems based on the AMD Athlon XP 2000+ and the Intel Pentium 4 2.0A processors. When compared to the SYSmark 2001 benchmark results, the SYSmark 2002 benchmark results suggested a significant performance decline in the AMD Athlon XP 2000+ processor’s performance and a significant increase in Intel Pentium 4 2.0A processor’s performance. This prompted AMD to contact BAPCo to better understand the methodology generating the unusual results of SYSmark 2002. Internet Content Creation Office Productivity Relative Performance: Relative Performance: AMD Athlon XP 2000+ AMD Athlon XP 2000+ Compared to Intel Pentium 2.0A GHz* Compared to Intel Pentium 2.0A GHz* 10% 10% 6% 1% SYSmark 2002 SYSmark 2002 0% 0% SYSmark 2001 SYSmark 2001 -10% -10% -4% -19% -20% -20% *See backup slides 15 & 16 for software patch information and configuration details 4 SYSmark 2002 White Paper To AMD After AMD contacted BAPCo for a detailed explanation regarding the changes to SYSmark 2002, AMD received the SYSmark 2002 white paper which included the following information: • The application workload changed between SYSmark 2001 and SYSmark 2002. In other words, the contribution of each application to the overall score changed. • BAPCo explains the methodology used to derive their SYSmark performance “ratings:” “The overall response time for a category (Office Productivity or Internet Content Creation) is the average of all the operation response times in all of the applications that make up that category. The average response time for each of the two scenarios is then converted to a performance “rating…” The overall SYSmark 2002 rating is derived from the geometric mean of the two category ratings.” • The only application changes were version updates to Microsoft Word, Excel, PowerPoint, Outlook and Access and PhotoShop and Windows Media Encoder. However: • BAPCo’s white paper explanation didn’t satisfactorily address AMD’s concerns. The white paper only addressed the application-level details and ignored the crucial task- level details and relative contributions of those tasks to the final score. So AMD investigated further… 5 Bapco Application Weighting Changes 40% 35% 30% 25% 20% 15% 10% SYSMark 2002Office Productivity 5% SYSMark 2001 0% Microsoft Word Microsoft Excel Microsoft PowerPoint Microsoft Outlook Microsoft Access Dragon Naturally Speaking Source: Bapco Whitepaper: An Overview of SYSmark 2002 – www.bapco.com McAfee VirusScan Netscape Communicator WinZip Computing WinZip 35% 30% 25% 20% 15% Internet Content Creation 10% 5% 0% SYSMark 2002 SYSMark 2001 Adobe Photoshop Adobe Premiere Macromedia Dreamweaver MacromediaFlash Microsoft Windows® Media Encoder 6 SYSmark 2002 Methodology: Example numbers for illustration only Application Task Time Application The chart on the left is a simplified (sec) Contribution summary of the SYSmark methodology. These are not actual Dreamweaver 4 App loading 5.5 times or percentages. Rather the Dreamweaver 4 Add product – Slider Pro 3 10% numbers are used to demonstrate Dreamweaver 4 Add radio button #2 1.5 the methodology: Flash 5 Group 9 1. The methodology selects a Flash 5 Deselect 12 24% number of tasks to be Flash 5 Rotate 3 performed by each application PhotoShop 6.01 Filter SmartBlur 14 2. The time spent on each task is its contribution to the total PhotoShop 6.01 Filter GaussianBlur 7.5 time PhotoShop 6.01 ImageSizePercent 3 25% 3. The sum of the task times for PhotoShop 6.01 Color Balance 0.5 each application is the Premiere App loading 7.5 application’s contribution to 10% the total time Premiere Open file 2.5 Media Encoder 7.1 Encoding 30 4. Finally, the total time is 31% converted to a performance Media Encoder 7.1 App loading 1 "rating" Totals 100 100% For example, the sum of the PhotoShop task times is 25 seconds. Because the total task time is 100 seconds, PhotoShop’s workload contribution to SYSmark 2002 Internet Content Creation is 25%. Similarly, the workload contribution of Filter SmartBlur to SYSmark 2002 is 14% of the total time (and 56% of the Photoshop application time). 7 AMD’s Findings: SYSmark 2002 Task-Level Patterns After further comparing SYSmark 2001 to SYSmark 2002 at the task- level, AMD found the following patterns: • Removing tasks that favor AMD • Adding tasks that favor Intel • Inflating workload contributions by repeating tasks that favor Intel The BAPCo white paper explanation of AMD’s SYSmark 2002 concerns ignored the task-level details and related contribution at the heart of the unusual results. This pattern was found across applications in both Internet Content Creation and Office Productivity. 8 AMD’s Findings: PhotoShop SYSmark 2001 Filters SYSmark 2001 used 13 different 8 FilterEmboss_Fade FilterOther_MaximumValue filters, one time each. The AMD Athlon FilterRender_LightingEffects… XP 2000+ outperforms the Pentium 4 FilterSharpen_SharpenEdges 2.0A on 8 of those 13 filters. FilterSketch_WaterPaper AMD Athlon XP FilterStylize+WindStagger… advantage FilterWaterPaper_Fade SYSmark 2002 uses 3 FilterWind_Fade SYSmark 2002 Filters Filter Artistic_Underpainting Filter Artistic_Underpainting filters from SYSmark Intel Pentium 4 FilterBlur_GaussianBlur Filter Artistic_Underpainting 2001 and 3 new advantage FilterSketch_Charcoal Filter Artistic_Underpainting filters, multiple times FilterSharpen_UnsharpMask Filter Artistic_Underpainting 5 FilterStylise_Emboss FilterBlur_GaussianBlur for a total of 20 runs. FilterBlur_GaussianBlur All 8 of the AMD FilterBlur_GaussianBlur Athlon XP 2000+ BAPCO selected only 3 of FilterBlur_GaussianBlur advantaged filters the original 13 filters from FilterBlur_GaussianBlur FilterSketch_Charcoal from SYSmark 2001 SYSmark 2001 to use in FilterSketch_Charcoal were removed from 12 FilterSketch_Charcoal 2002. Those 3 filters SYSmark 2002, favored the Intel Pentium FilterBlur_SmartBlur FilterBlur_SmartBlur resulting in the Intel 4 2.0A GHz and are FilterBlur_SmartBlur Pentium 4 2.0A GHz FilterBlur_SmartBlur repeated to account for 12 outscoring the of the 20 times that a filter FilterBlur_SmartBlur FilterRender_LensFlare AMD Athlon XP on all is run in Sysmark 2002. FilterRender_LensFlare 20 FilterOther_HighPass SYSmark 2002 filters. 9 AMD’s Findings: Flash Task Time Difference between AMD Athlon XP 2000+ and In SYSmark 2001, Intel Pentium 4 2.0AGHz “Step Frame,” which the AMD Athlon XP step frame deselect other excelled at, accounted for 211 of 241 flash 40 tasks (88%), but was SYSmark 2001 SYSmark 2002 completely removed in AMD Athlon XP advantage 8.669 SYSmark 2002. 0 -4.659 Intel Pentium 4 -3.668 -29.895 advantage In SYSmark 2002, the tested file size greatly -40 increased from 118 kb -32.81 to 2.662 MB. Time (seconds) -80 10 AMD’s Findings: Excel In SYSmark 2002, Task Time Difference between sort is now AMD Athlon XP 2000+ and responsible for Intel Pentium 4 2.0AGHz over 90% of the application time. App loading sort charting other 5 SYSmark 2001 SYSmark 2002 AMD Athlon XP 0.458 advantage -0.243 -0.094 0.498 0 -0.066 Intel Pentium 4 -0.029 advantage -0.021 -5 -11.220 -10 Time (seconds) -15 11 AMD’s Findings: Access The 14 Access tasks where the AMD Task Time Difference between Athlon XP 2000+ AMD Athlon XP 2000+ and achieved the biggest Intel Pentium 4 2.0AGHz advantage over the Intel Pentium 4 2.0A AMD Athlon XP advantaged tasks GHz in SYSmark 2001 Intel Pentium 4 advantage tasks were completely Top 14 Athlon XP advantage tasks removed in SYSmark 2002. 40 SYSmark 2001 SYSmark 2002 Those 14 tasks 30 represented over 75% of the 20 application time 27.204 and 90% of the advantage the Time (seconds) 10 AMD Athlon XP AMD Athlon XP advantage 0 3.821 1.942 2000+ held over 0 -0.176 the Pentium 4 Intel Pentium 4 -1.331 advantage 2.0A GHz. -10 12 AMD Decisions AMD’s analysis raises serious questions regarding changes to the selection of tasks and application contribution in the methodology. BAPCo’s white paper response to AMD did not resolve the task-level questions raised by our analysis. Therefore, AMD plans to: • Join BAPCo in order to more fully understand