SPEC OSG Mailserver Subcommittee SPECmail2008 Benchmark Architecture White Paper Revision: v1.1 Date: 2 June 2008 1. Introduction 1.1 Overview SPECmail2008 is a software benchmark designed to measure a system's ability to act as an enterprise mail server servicing email requests, based on the Internet standard protocols SMTP and IMAP4. The benchmark concentrates on the workload encountered by corporate mail servers, with an overall user count in the range of 150 to 10,000 (or more) users. It models IMAP business users accessing IMAP servers over fast local area networks (LAN) instead of broadband, WAN or dialup access speeds. SPECmail2008 has been developed by the Standard Performance Evaluation Corporation (SPEC), a non-profit group of computer vendors, system integrators, universities, research organizations, publishers, and consultants. This paper discusses the benchmark principles and architecture, and the rationale behind the key design decisions. It also outlines the workload used in the benchmark, and the general steps needed to run a benchmark. However those aspects are covered in more detail in other documents. 1.2 Organization of this Paper Chapter 2 discusses the basic goals and non-goals of the benchmark. Chapter 3 introduces the performance metric of SPECmail2008 – IMAP sessions per hour - and how it relates to the transaction mix imposed on the system under test.. Chapter 4 explains the benchmark workload - how it was derived, how it translates into configuration parameters for the benchmark tool and size calculations for planning a benchmark, and how it relates to the benchmark metric. Chapter 5 discusses some detail of aspects of the workload generation, namely the exact workload put on the server, and how the benchmark simulates communication with remote mail servers. Chapter 6 defines the quality of service requirements of this benchmark. Chapter 7 lists the references and sources (not cited elsewhere). 1 1.3 Related Documents SPECmail2008 Run and Reporting Rules Workload Analysis for Enterprise Mail Servers SPECmail2008 User Guide SPECmail2008 Sample Result Disclosure SPECmail2008 FAQ All documents can be obtained from the mail server working group's home page http://pro.spec.org/private/osg/mail_server. 1.4 Run and Reporting Rules The Run and Reporting Rules for the SPECmail2008 benchmark are spelled out in a separate document. They ensure execution of the benchmark in a controlled environment. The goal is repeatability by third parties with reasonable resources and knowledge. The rules maximize the comparability of results, leveling the playing field as much as possible. They also define which information needs to be included in published results, and which supporting information needs to be submitted to the SPEC community for potential review. Under the terms of the SPEC license, SPECmail2008 results may not be publicly reported unless they are run in compliance with the Run and Reporting Rules. Results published at the SPEC web site have been reviewed and approved by the SPEC Mail Server committee. For more information on publishing results at the SPEC web site, please send e-mail to: [email protected]. The Run and Reporting Rules may be found on the SPEC web site; they are also part of the SPECmail2008 distribution kit. 2. Design of the SPECmail2008 Benchmark SPECmail2008 benchmark tests the capacity of a system as an e-mail service that processes requests according to the Internet standard mail protocols SMTP (RFC 821) and IMAP4 (RFC 2040). The SMTP protocol is the standard for sending e- mail from clients (users) to servers and between e-mail servers. The IMAP4 protocol allows users to access and retrieve messages from their message store. The mail server can consist of one host or a group of hosts that act as a single, logical entity – usually represented by a single e-mail domain. The SPECmail_MSEnt2008 metric’s user model describes a corporate employee that uses one of the popular IMAP4 clients to access either a mail server located within a local area network (LAN), or an outsourced e-mail service across a high- speed network connection (MAN). The details of an enterprise-type user's behavior will be discussed in a later section in this paper that covers the SMTP and IMAP work load profiles. In this benchmark, both the Mail Server User behavior and the IMAP4 e-mail client software vary greatly. Therefore, it is very important to identify and distinguish 2 actual IMAP4 client’s human initiated actions from automated actions performed on behalf of each user. The specific combination of these behavior types determines how many users a mail server can handle. SPECmail2008 simulates the work loads of four types of IMAP4 e-mail clients, defined in a fixed proportion across the user population. The benchmark observes the mail server behavior under that load. It enforces the adherence to a required level of quality of service. The goal is to simulate realistic mail server operation, and maximize the usefulness of the benchmark results as guidelines for actual sizing decisions. 2.1 Requirements and Goals The key goal of SPECmail2008 is to show mail server performance in a realistic context. This means Appropriate Mail Store A mail server which handles the transaction load of a corporate user base needs to hold the mail data for the same amount of users. This includes a mailbox per user, as well as a defined number of folders and messages for each user. Mail Store Folder Structures The benchmark includes the construction and pre-population of multiple folders (IMAP mailboxes) and subfolders as part of the compliance requirements. Since IMAP is a server-centric storage service, access and manipulation of folders plays a critical part of the overall workload. Message MIME Structures The benchmark includes the construction and manipulation of e-mail messages with both simple and more complex MIME (messages with attachments) structures. Unlike POP3 servers, IMAP clients understand and can manipulate multi-part MIME messages. Arrival Rate Message activity requests are being issued against the server according to behavior patterns and arrival rate modeled after real world observations. Access over Local Area and/or All users will access the IMAP and SMTP servers via internal corporate or Broadband Networks broadband network speeds. Workload According to Real The simulated workload (such as messages sent and received per day, the World Data message size distribution, the folder structure distribution, the mailbox access frequency, message manipulation) is based on measurements derived from multiple data sources. Peak Hour Simulation Mail traffic is distributed unevenly over the day. Sizing must focus on coping with the peak load. Therefore, the benchmark mail server load simulates the peak hour of the day. Realistic Operation The mail server is required to operate realistically, e.g. perform at least a defined level of logging which many ISPs would use in practice. As is the rule in all SPEC benchmarks, no benchmark-specific optimizations are allowed. 3 2.2 Excluded Goals Explicitly excluded goals and limitations in the scope of SPECmail2008 are: Restricted client IP Range The benchmark does not require that there be 100's of small, remotely connected email clients. Instead, it allows simulation of the email clients on a small number of server-sized, locally connected client systems, in order to make the benchmark execution practical. Administrative Overhead The benchmark does not include administrational activities like on-line backup and account provisioning. These are hard to standardize, and - more importantly - they do not necessarily happen during the peak hour of daily operation. Provisioning Overhead Besides provisioning, the benchmark does not include modification to account data or deletion of accounts. These activities, although commonplace in the day-to-day operation of a mail server, are seen as testing the directory service rather than the mail server itself. SPECmail2008 emphasizes mail server benchmarking, and includes a directory server in the system-under-test only to the minimum extent necessary to handle normal mail flow. Pure Relay SMTP Traffic The benchmark does not cover relay operation (forwarding of incoming SMTP traffic to other, remote MTAs). It does include simulation of messages from local users to remote MTAs, as well as incoming mail from remote MTAs. Relay operation is generally not allowed on enterprise e-mail systems. High Availability Overhead There is no requirement for high availability or disaster recovery measures; this was beyond the scope of the benchmark. Extra Content The mail server is not required to perform any extended features, like virus Filter/Processing, Security scanning, spam filtering and other security measures. This was beyond the scope of the benchmark. SMTP Streamlining SMTP sessions can normally send one or several messages per session. The benchmark restricts itself to one message. The collected SMTP log data shows that this behavior dominates internally and externally generated e- mail traffic. 3. Benchmark Metric: SPECmail_MSEnt2008 SPECmail2008 Enterprise IMAP4 Sessions Per Hour The basis of the benchmark’s performance metric is capacity at acceptable quality of service (QoS). The benchmark determines Acceptable QoS by measuring the interactive response time of the mail server to each protocol step (see Chapter 6). The boundary where a one or more critical states exceeds the Acceptable QoS determines how many users the SUT supports. The IMAP protocol allows many combinations of session behavior and duration – very unlike a typical POP3 session used in SPECmail2001. Next, IMAP clients use more than one connection into an IMAP server to perform different tasks. Lastly, each IMAP user generate many IMAP sessions – both in parallel and serially over the peak hour period. The number of extra sessions varies by IMAP client 4 software. Therefore the number of concurrent IMAP4 sessions is often some multiple of the actual active IMAP users. The actual count is determined by both active users and the specific distribution of IMAP client types.