EEMBC JOURNAL AUTUMN 2005

JOURNAL

INSIDE THIS ISSUE: FROM THE PRESIDENT FROM THE 1 PRESIDENT New Subcommittee Targets Network Storage FROM THE LAB 1

NEWS BRIEFS 2

FOR MEMBERS 2 If you're a regular reader of EE ONLY Times, Communications Business Daily, Electronics Weekly, or various other publications that progression toward system- The second respect in which cover the network storage indus- level benchmarks. Its kernel we’re making a departure is in try, chances are you’re already will address the per- the genesis of the EEMBC Net- aware that EEMBC has formed formance of CPUs and special- work Storage subcommittee that a new subcommittee that will be ized engines used in network is developing StorageBench. For developing benchmarks to pre- storage applications, while a the first time in the Consortium’s dict the performance of CPU second benchmark will measure history, the impetus to develop a subsystems in network storage system-level throughput, running new suite is coming directly from applications. This is exciting data that has been shaped to be the community of embedded news for several reasons. For representative of real storage processor customers, and in one thing, the new Storage- system traffic through the chip David Solina of Adaptec we have Bench™ suite will fulfill a major and monitoring its performance. (continued on page 3) need for manufacturers in major need for manufacturers in the network storage space. It will give them the industry’s first From the Lab certifiable benchmarks that can measure items such as raw data Stabilizing Benchmark Scores movement, I/O per- formance, and the effects of in EEMBC: Software “Jitter” core-to-memory latency across a variety of storage-related plat- Alan R. Weiss, EEMBC Certification Laboratory forms. These are all useful things to know with respect to network attached storage (NAS), storage There is no such thing as Cygwin environment or under area networks (SANs), virtual “software jitter” in benchmarking, or Solaris before they try tape libraries (VTLs), and direct but the term is a very good de- porting to an embedded target. attached storage (DAS) RAID scription for what you often see systems. when you run benchmarks on a In these cases, what you really personal computer with an OS observe is non-deterministic StorageBench is the 8th or 9th such as Linux that does not run variance. There are a number of (depending on how you count) in real time. It might as well be causes of this variance, each application-focused benchmark called software jitter, though, with its own solution: suite that EEMBC has developed. because the variance in bench- • Not running the benchmark But it also represents a “first” in mark scores is fairly high, even long enough to take into ac- several respects, and that’s an- after you control for the environ- count the granularity of the OS other reason to be excited. For ment. In EEMBC, members often www.eembc.org www.eembc.org for the in benchmark latest scores timers, i.e. a call to clock() one thing, StorageBench repre- download and run software on an sents the latest step in EEMBC’s PC under Windows in a (continued on page 3) Page 2 EEMBC Journal Autumn 2005

NEWS BRIEFS

EEMBC’s newest “The Embedded held October 13 in Sindelfingen, member is Benchmark Consortium (EEMBC) is Germany. Between these engage- CodeSourcery, a taking on these challenges. With ments, Levy and Greg Crouch of provider of devel- three major initiatives, the group National Instruments presented opment tools, hopes to create new, open and “Evaluating and Selecting software test licensable benchmarks that will be Processors/ for solutions, and predictive for the multicore SoCs Real-Time Applications” at the Real- custom software and sophisticated applications that Time Automotive Seminar on development characterize the current market.” November 3 in Dearborn, Mich. Mark Mitchell of CodeSourcery services for semi- Read more at Levy is next slated to present at conductor and www.eetimes.com/showArticle.jhtm Embedded World 2006, being held operating system vendors. Joining l?articleID=174900199 . February 14-16 in Nuremberg, the Consortium in the third-party Germany. Topic for this upcoming tool member category, Granite EEMBC is accepting job applica- presentation will be “The Changing Bay, Calif.-based CodeSourcery’s tions for the full-time position of Landscape of products and services include Director of Software Engineering. Selection,” an exploration of the support and optimization for Responsibilities include benchmark performance variations among Sourcery G++, CodeSourcery’s source code development and different ARM7 microcontrollers. project management, technical version of the GNU Toolchain. “Our www.embedded-world-2006.de customers—many of whom are also support, maintenance and develop- EEMBC members—have made it ment of technical documentation, clear that the EEMBC benchmarks benchmark score certification, are one of the key metrics they use and technical conference presenta- when evaluating ,” says tions. optimization Mark Mitchell, President and experience very helpful. The Chief Sourcerer of CodeSourcery. successful candidate must have MEMBERS ONLY “We're very excited to be joining exceptional interpersonal commu- the EEMBC community and look nication and presentation skills. The next meeting of the EEMBC Board of Fluent in several languages is a forward to further improving our Directors will take place January 10-11, 2006 products and services by using this plus. To apply, send a resume to powerful tool.” Markus Levy, EEMBC President, at the Wynn Las Vegas. To reserve, call at [email protected]. 1-866-770-7555 and identify yourself as a EEMBC’s GrinderBench Web site (www.grinderbench.com), which EEMBC’s academic licensing member of the EEMBC Quarterly Meeting group. allows users to test the per- program has been expanding at Further information on the venue is available unprecedented rate. Latest to join formance of their mobile phones at www.wynnlasvegas.com. Details of the and PDAs, now allows users to up are faculty at the universities of automatically calculate their single- Nebraska, Pittsburgh, Singapore, schedule and agenda are available from the number Grindermark scores. The Texas (Austin), and Tubingen main page on the members-only area of the (Germany). EEMBC President same mechanism allows EEMBC to EEMBC Web site. supplement score data submitted Markus Levy reports much interest by members with scores from the in the OEM community as well in A new member resource for information on larger community of mobile phone licensing EEMBC’s benchmark and PDA users. software. EEMBC and its benchmarks is “Conference Presentations for Use by Members” on the “. . . [B]enchmarking embedded Recent speaking appearances by systems has always been difficult. EEMBC President Markus Levy members-only area of the EEMBC Web site. As those simple MPUs of yore have included a keynote speech at Here you’ll find a number of EEMBC Powerpoint evolved into complex, multitasking HiPEAC 2005, the International presentations from recent conferences, saved systems-on-chip — often with hard- Conference on High Performance ware accelerators, shared on-chip Embedded Architectures and to ZIP archives, that you can download to your memory structures and buses, and Compilers held November 17-18 in computer and use to add EEMBC information intelligent — the simple Barcelona, a presentation at the to your own presentations. Look for the new question of whether a given chip Embedded Systems Show in can do the job has become a huge Birmingham, England, and a key- section under “Marketing and Promotional issue,” writes Ron Wilson in the note speech at the Chips and Tools Resources.” December 5 issue of EE Times. conference Page 3 EEMBC Journal Autumn 2005

EEMBC Establishes New Subcommittee NEWEST BENCHMARK (continued from page 1) SCORE REPORTS

ST231 300 MHz our first subcommittee chair In this regards, I invite any whose company is also an em- vendor involved in designing Production Silicon bedded processor customer. some aspect of storage-based Automotive platforms to contact me and Out-of-the-Box In fact, many of our members consider joining our organization Consumer have customers in the network to get directly involved in the Out-of-the-Box storage market, and with this definition, development, and new benchmark suite we aim to Networking V 1.1 testing of this interesting bench- give them a powerful tool to ana- Out-of-the-Box mark suite. lyze and validate new processor Office Automation architectures and optimize prod- I wish you a joyous Out-of-the-Box ucts for storage applications. We holiday season and Telecom are particularly fortunate to have look forward to Out-of-the-Box the participation and support of sharing with you David Solina and his colleague many exciting Kimberly Robinson at Adaptec, developments in and all of us are looking forward EEMBC in 2006. to maintaining the momentum of this effort and ensuring that the benchmarks we develop are fully relevant to the needs of all storage product manufacturers. Markus Levy

From the Lab

Stabilizing Benchmark Scores in EEMBC: Software “Jitter” (continued from page 1)

• or gettimeofday() • The OS is managing memory (firmware) and power manage- function and processes by swapping or ment systems that go behind the paging (which is what a good back of the OS to throttle the Solution: Since EEMBC’s signifi- OS should do) processor. Some code-morphing cant figure of merit is iterations x86 processors seem to do this per second, increase the number Solution: The same as above - kill as part of their strategy to save of iterations for each benchmark unnecessary processes, threads, power, for example. Look for both so that they execute for at least 3 and device drivers. Make the BIOS settings as well as power to 5 seconds. On a 32-bit or 64- system as quiescent as possible. management device drivers. bit machine, there is little danger Do NOT move the mouse! Do not of clock overflow errors. I think cause external interrupts. In the end, repeatability relies you’ll find that variance goes upon scientific control of all • The clock() and get- down substantially by doing this. variables, something ECL has timeofday() functions may vast experience in doing. This is • OS system services and device be broken, returning incorrect another good reason why your drivers are not quiescent values company should submit its materials to ECL for certification. Solution: Make sure you are run- Solution: Fix the bug. Some early Even if you don’t publish the ning the benchmarks with as few versions of Linux had all sorts of esults publicly, we can still help other processes and threads weird things happening. Some you in numerous ways, improved running as possible. Judicious embedded versions of Linux still repeatability being one of them. use of top(1) and kill(1) have incorrect coding of the inter- on Linux or Unix machines works face from OS calls to the RTC. wonders here. Kill all non- • essential processes and threads Inexplicable variances between and by all means kill X windows runs system on a Unix or Linux Solution: When all else fails, it is machine. valid to suspect the BIOS

EEMBC is a registered trademark of the Embedded Microprocessor Benchmark Consortium. All other trademarks appearing herein are the property of their respective owners.