EEMBC JOURNAL WINTER 2005

JOURNAL

INSIDE THIS ISSUE: FROM THE PRESIDENT FROM THE 1 PRESIDENT EEMBC Celebrates Launch of Digital FROM THE LAB 2 Entertainment Benchmarks and Scores NEW 2 SCORES and picture complexity, allowing quality measure. This quality BENCHPRESS 3 us to demonstrate how a digital measure is based on a signal to entertainment system handles noise ratio (PSNR) to provide FOR MEMBERS 3 ONLY different workloads. In other “an estimate of the quality of a words, one of the video streams reconstructed image compared NEWS BRIEFS 3 will be applicable to the small with an original image” (a sum- screen on a mobile phone, while mary of PSNR is available here). It’s been a long time coming . . . another video stream is repre- PSNR is a fairly common industry EEMBC has released the official sentative of the quality you’d find practice that ECL integrated version of its new Digital Enter- on a high-end set-top box. into our Test Harness to provide tainment benchmark suite a uniform approach that all (DENbench™), and we also have Another interesting twist that of EEMBC’s members could four influential members that we’ve applied to these bench- adhere to. have certified and published marks (specifically, the MP3 scores already on these new player and video encode/decode (continued on page 4) benchmarks. Before digging into benchmarks) is the use of a the details of these benchmark scores, corny as it may sound, I am very proud to be able to offer this suite to EEMBC’s member- ship. It is, by far, the most com- prehensive, sophisticated, and challenging benchmark suite we have offered to date. The suite’s 4 FREE certifications from ECL if you more than 60 discrete tests include MPEG, MP3, JPEG, and reserve by January 31. cryptography. And as an added ECL, the EEMBC Certification Lab, will provide EEMBC Board bonus, our cryptography bench- members with up to 4 free out-of-the-box or 2 free full- marks have already secured a fury certifications provided reservations are made by January place in EDN’s top 100 technol- 31, 2005 and members agree to publish the resulting scores. ogy products of 2004! Subcommittee-only members are entitled to one free out-of- These benchmarks offer an the-box certification. innovative departure from Certified benchmark scores add a distinct note of credibility previous EEMBC practice with to new product launches . . . or generate new buzz around the use of multiple datasets. For existing products. Use your free certifications to add to the example, the MPEG-4 encode products in your portfolio for which you can offer objective benchmark has five unique performance measurements, or run benchmarks on video streams that get fed in. the same product again with a different , Each video stream represents a clock, or bus speed. All at no charge www.eembc.org www.eembc.org for the in benchmark latest scores different bit-rate, screen size, from ECL.

For further information, please E-mail [email protected]. http://www.ebenchmarks.com Page 2 EEMBC Journal WINTER 2005

NEWEST BENCHMARK SCORE REPORTS

IGNITE 2FX 600 MHz TC1130 150 MHz TMS320C64x 1 GHz Simulation Production Silicon Production Silicon Out-of-the-Box Out-of-the-Box Out-of-the-Box Consumer Automotive/Industrial Telecom

C Optimized Telecom Visit the new DENbench web page at www.eembc.hotdesk.com/digital_entertainment.html

Assembly Optimized Telecom ADSP-BF533 AMD Geode MPC7447A IBM 750GX Processor Name-Clock 594 MHz NX1500@6W - 1 GHz 1.4 GHz 1 GHz MPEG Decodemark™ N/A 785.1 1506.3 1054 MPEG Encodemark™ 355.5 587.4 1281.5 967.9 Cryptomark™ N/A 509.3 1263.3 903 Imagemark™ N/A 918.9 1709.4 1090.4 MPEG2 Encode™ N/A 30.2 67.2 N/A (Floating Point) DENmark™ N/A 131.7 257.6 173.6

From the Lab The Value of Reference Platforms EEMBC Calendar for Comparison Purposes February 22–24, Nuremberg EEMBC exhibits at Embedded World 2005 (Nuremberg, Germany) within stand 10-321. Alan R. Weiss, EEMBC Certification Laboratory (ECL, LLC) For show information, visit www.embedded-world- 2005.de.

Benchmarking is inherently a figure of merit. EEMBC uses February 23, Nuremberg comparative, and competitive, iterations per second as a figure of Markus Levy presents “Analyzing the Tradeoffs Be- activity. While absolute scales are merit for many of its performance tween Performance and Power When Designing Em- fairly common in science (for benchmarks, but these must be bedded Applications” at 4:30 p.m. as part of the Em- example, the Kelvin scale for translated into a single number, in bedded World 2005 conference in Nuremberg. For temperature), business and a range easily understood. A score conference details, visit www.elektroniknet.de/ engineering people like to measure of 3,238,560 on one benchmark termine/ew2005/index.htm. directly against others. So when kernel, followed by a score of 594 EEMBC develops a new benchmark on another kernel in the same March 20-22, Austin suite, until there are at least a few suite, results in some funny At the 2005 IEEE International Symposium on Perform- benchmark scores available, it is arithmetic when you try to average ance Analysis of Systems and Software (ISPASS), Mar- difficult to know a priori if a score them together. kus Levy will deliver the keynote address “EEMBC and of, say, 25.5 DENmarks is good — Reference platforms can solve this the Purposes of Embedded Processor Benchmarking.” or very low. How does this compare problem by offering a measuring For location details and further information, visit against the fastest processors on stick against which one can gauge www.ispass.org/ispass2005. the market? How does this another processor, and in helping compare to other processors in its to calibrate the single-number con- April 18, San Jose class (price, power consumption, solidated score(s) (such as the Markus Levy kicks off the plenary session of the Port- temperature range)? Telemark, Consumermark, and able Power Developers Developer’s Conference at 10 Engineers may want to analyze OAmark) that EEMBC reports for a.m. with “Analyzing Power Efficiency in Portable Appli- each and every benchmark score in each of its benchmark suites. But it cations.” Further information is available at a given benchmark suite, but is no simple matter to decide which www.darnell.com/ppdc. marketeers seem to want a single (continued on page 4) Page 3 EEMBC Journal WINTER 2005

Visit the EEMBC “In the News” page to read the latest in editorial coverage.

Automotive real-time benchmarks, by Markus Levy. Embedded Computing Design, December 2004.

EEMBC Adds a Metric, by Don Tuite. Electronic Design - Powerful Ideas for Design. November 29, 2004.

Microprocessor consortium to add energy spec. EE Times, Novemnber 9, 2004.

Editorial: Can benchmarking be rational?, by Kevin Krewell. Watch, September 24, 2004.

Networking benchmarks score Freescale and IBM chips, by Graham Prophet. EDN Europe, September 2, 2004.

NEWS BRIEFS Tokyo-based IPFlex Inc. has joined EEMBC as a full member of the Consortium’s Board of Direc- Members Only tors, with full voting rights on all major issues decided by the consortium, plus full access to all Following a series of briefings with analysts and editors in of the EEMBC benchmarks. IPFlex’s flagship product, the DAPDNA- the U.S., Europe, and Asia, EEMBC announced publication 2, is a dynamically reconfigurable processor with the ability to of the first scores based on its DENbench Digital Entertain- switch functions by dynamically and instantly changing the circuit ment benchmarks on January 24 (see scores on page 2). configuration within the chip. The processor is for commercial use, “EEMBC’s release of these new benchmarks and the publi- suitable for the areas of telecommunications and security where cation of these scores from AMD, ADI, Freescale, and IBM mass capacity high-speed processing is required, and for industrial is very timely given the rapid growth of the digital entertain- and medical image data processing. www.ipflex.com. ment market,” said industry analyst Will Strauss, founder and president of Forward Concepts. “These benchmarks EEMBC's Cryptography benchmark, part of the new provide a comprehensive interpretation of a proces- Digital Entertainment benchmark suite, was among the sor’s system performance running a wide variety of Hot 100 Products of 2004 announced in the Decem- digital entertainment code, and they have been ber 17 issue of EDN. "Although how innovative or 'hot' constructed to allow apples-to-apples comparisons a product is never ensures market success, these among a wide range of processors.” products are the 100 we think are the most promising toward that end among the thousands that vendors At its meeting in Menlo Park, Calif. on December 7, introduced in 2004," said John Dodge, EDN editor in the EEMBC Board of Directors voted to create a Multi- chief. The award was made in the Embedded Tools processing Subcommittee to address benchmarking category. www.edn.com/article/CA486570.html of multiprocessing/multithreading (MP/MT) devices. The group, headed by John Goodacre of ARM Ltd. and Mike EEMBC mourns the loss of Kaivalya Dixit, long-time president of Uhler of MIPS Technologies, discussed the approach that Standard Performance Evaluation Corp. would be taken in developing the new suite, which will ini- (SPEC), who died November 22 at age 62. Dixit tially focus solely on symmetric multiprocessing (SMP) had headed SPEC since 1990. A 30-year vet- implementations, where the software assumes homogene- eran of the computer industry, Dixit worked ity across processing elements, threading is the technique most recently at IBM in Austin, Texas. He also to express concurrency, and an OS or RTOS is available to held engineering, marketing and management abstract the hardware into an agreed-upon software positions with Sun Microsystems and Ford threading model. Consumer, mobile, and PDAs were Aerospace. Dixit is survived by his wife Evelyn mentioned among the applications into which the devices of Austin, and son Raj of Bakersfield, Calif. proposed for benchmarking are being deployed. Memorial contributions can be made in mem- ory of Kaivalya Dixit to the American Heart Association or the The next EEMBC Board of Directors meeting is tentatively American Diabetes Association. scheduled for March 23-24 in Austin, Texas. Location www.spec.org/spec/kaivalya/contributions.html details will be announced in February. Page 4 EEMBC Journal WINTER 2005

The Value of Reference Platforms for Comparison Purposes (Continued from Page 2)

processor, on which board, and For the second generation of bed to develop and test the high-performance 32-bit and 64- at what clock speed, should Consumer benchmarks (called code (using various C compil- bit processors. serve as the reference platform Digital Entertainment, or DEN- ers), along with a half-dozen for a given benchmark suite. bench), Advanced Micro Devices other platforms. Building the Networking Version 2 (IPmark One issue is finding a reference allowed the EEMBC Certification code was thus a matter of sim- and TCPmark), as well as the platform that can serve as a Laboratory (ECL) to use its ply doing what every EEMBC soon-to-be-released Office Auto- typical example of the expected GeodeNX1500@6W (1 GHz) as member can do: downloading mation Version 2 (featuring performance range. Another a baseline platform and to share the code from the EEMBC web- Ghostscript), will use a similar issue is the availability of a vol- these scores with EEMBC mem- site, unzipping it, making sure a technique (although reference unteer member company to bers. The Geode is a midrange C compiler was installed, and platforms haven’t been selected offer its platform as a reference and highly efficienct processor typing make. The new makefile yet for those benchmark suites). platform. targeted for set-top boxes, Inter- system developed by ECL for For 2005, EEMBC will continue net appliances, and thin clients, EEMBC did the rest, including to use the concept of reference Fortunately, EEMBC has had and it is commonly available in emitting comma-separated files platforms as ECL develops Auto- some volunteers. the market. By not using a very easily imported into Open Office motive/Industrial Real Time high-end PC processor such as or Microsoft Excel workbooks. Version 2 and other benchmark For the GrinderBench suites. benchmark suite, a Sharp AMD Athlon-64 or an Pen- Then came the hard work. Con- Zaurus running Embedix 1.0 on tium 4, a message was sent that sumer Subcommittee members, And what did AMD get for its top of an Intel XScale 206-MHz the purpose of this exercise was led by Sergei Larin of Freescale troubles? A free certification and processor was used to help cali- in calibrating the benchmarks Semiconductor, worked to de- publication of scores, and the brate the GrinderBench score. for a specific purpose — not in velop the arithmetic necessary knowledge that they have con- This was appropriate, because blowing away the field. to calibrate the DENmark tributed to the development of PocketPCs, PalmOS devices, scores. Many experiments were an extremely important new and modern mobile phones ECL had little trouble producing tried to ensure that DENbench benchmark suite. were expected to be within a the scores: when it develops the would be as applicable for range 10x slower to 10x faster benchmark code for EEMBC, it lower-end 16-bit fixed-point than this platform. regulary uses PC's as a test processors as it would be for

FROM THE PRESIDENT (continued from page 1) Okay, now who are those bold score provides a single-number 1.4 GHz, this processor cranks from being glowing hot. These and progressive members who for the en- through video frames faster devices, especially the BF533, have published the first set of tire DENbench suite. than a sous chef chops vegeta- are targeted at low-power em- scores? First is AMD with its 1- bles. And this doesn’t account bedded applications with pas- GHz AMD Geode NX1500. Next As we’ve seen with EEMBC’s for the additional benefits that sive cooling. is with its 600- other benchmarks, system can be provided through the It’s also possible to make an- MHz ADSP-BF533, based on the designers will interpret the DEN- processor’s AltiVec engine. This other comparison by applying a Blackfin architecture. There’s bench scores depending on level of performance makes the performance per dollar metric. I also the PowerPC-based 1.4- how they prioritize design con- MPC7447A perfect for high-end leave it to you to do that math GHz MPC7447A from Freescale. siderations such as speed, effi- encoder applications (i.e. this and draw your own conclusions. And last but not least is IBM ciency, power consumption, die is the machine that would en- For EEMBC, though, the bottom with its 1-GHz IBM 750GX size, and price. The various code the video streams before line is that there are really four PowerPC machine. These proc- consolidated scores that the they are blasted out over the winning processors here. Each essors represent a wide range benchmarks produce become Internet). has specific features that target of performance, power, and most useful, in fact, when different markets, and together price characteristics, and there- used to calculate how much a Now if we break it down with they are an excellent showcase fore help to validate the effec- processor’s performance ‘costs’ an execution efficiency rating for the ability of DENbench to tiveness of DENbench. in terms of several different (i.e. performance/MHz), the compare processors in a wide metrics. Freescale device still does quite For each processor measured well (due to its beefy microarchi- range of applica- tions, from MP3 with DENbench, EEMBC can Let’s look at a few examples of tecture), although the IBM players to mobile report individual results for up to how the different metrics can device surpasses it on MPEG phones to set-top 69 benchmark algorithms and apply. Referring to the bench- encode algorithms. On the other boxes. associated datasets, as well as mark scores from the proces- hand, look at it from the point of a series of consolidated scores sors listed above (available at view of , that provide a snapshot of www.eembc.org), the and the Analog Devices and performance in specific test MPC7447A is the raw perform- AMD processors really start to groups. An overall DENmark™ ance leader. Running at shine. And I don’t mean ‘shine’ Markus Levy

EEMBC is a registered trademark of the Embedded Microprocessor Benchmark Consortium. All other trademarks appearing herein are the property of their respective owners.