SPECmail2008 User Guide

Version 1.0.2
Last modified: 27 May 2008


1     Introduction
1.1   Terminology
1.2   Overview
2     How to install SPECmail2008
2.1   Before you begin - a pre-installation checklist
2.1.1 Network Configuration
2.1.2 Directory Services
2.1.3 Mail Domain Names
2.1.4 Disk Space Allocation
2.1.5 Mailbox Creation
2.1.6 Client Configuration
2.1.7 Synchronizing the Clocks of the Servers and Clients
2.2   Setting Up the Server
2.2.1 Installing your Server
2.2.2 Testing your Server
2.2.3 Creating User Accounts on the Server
2.3   Setting Up the Clients
2.3.1 Installing Java
2.3.2 Installing the SPECmail2008 Software
2.4   RC Files
2.4.1 Configuration Parameters
2.4.2 System Under Test (SUT) Description
2.4.3 Trace Parameters
2.4.4 Fixed Parameters
3     How to Run SPECmail2008
3.1   Starting the Load Generators
3.1.1 Starting the Load Generators on UNIX
3.1.2 Starting the Load Generators on MS Windows
3.2   Starting the Mail Sink
3.3   Starting the Benchmark Manager
3.3.1 Options
4     Results
4.1   Generating a SPECmail2008 HTML Report
4.1.1 In UNIX
4.1.2 In MS Windows
4.2   The HTML Results File
4.2.1 Results Block
4.2.2 Summary Results
4.2.3 System Descriptions
4.2.4 Configuration Diagram
4.2.5 Detailed Results
4.2.6 Notes/Tuning Information
5     Tips and Troubleshooting
5.1   Using Java Version v1.5
5.2   Benchmark Manager Only Prints Version Number
5.3   Connecting to Remote Clients Failed: InvalidClassException
6     Submitting Results
Appendix A – Configuration Parameters
Appendix B – System Description Parameters
Appendix C – Trace Parameters
Appendix D – Fixed Parameters

1. Introduction

SPECmail2008 is a client server benchmark for measuring mail server performance.

Software on one or more client machines generates a benchmark load for a System Under Test (SUT) and measures the SUT response times. A SUT can be a mail server running on a single system or a cluster of systems.

A SPECmail2008 'run' simulates a 100% load level associated with the specific number of users, as defined in the configuration file. The mail server must maintain a specific Quality of Service (QoS) at the 100% load level to produce a valid benchmark result. If the mail server does maintain the specified QoS at the 100% load level, the performance of the mail server is reported as SPECmail_MSEnt2008 IMAP sessions per hour. The SPECmail_MSEnt2008 IMAP Sessions per Hour metric reflects the unique workload combination for a SPEC IMAP4 user.

This document is a practical guide for setting up and running a SPECmail2008 benchmark. This user guide discusses some, but not all, of the rules and restrictions pertaining to SPECmail2008. Before continuing with the benchmark, we strongly recommend you read the complete SPECmail2008 Run and Reporting Rules contained in the kit. For an overview of the benchmark architecture, see the SPECmail2008 Architecture White Paper also contained in the kit.

1.1. Terminology

The System Under Test (SUT) is the hardware and software that makes up the mail server being tested by the benchmark. It includes the hardware system, disks, and network components used, as well as the OS and mail server software. The client machines (clients) running the load generators are not considered part of the SUT.

The Load Generator is the program that creates traffic (IMAP4 and SMTP) for the SUT. You can have as many clients running load generators as you wish, but you should have a minimum of two. One client will be the SMTP mail sink and one will be the benchmark manager.

The SMTP Mail Sink is a load generator that receives traffic from the SUT and that simulates a remote mail server. During the benchmark run, the load generators will generate mail for internal and external mail recipients. Mail intended for external mail recipients should be relayed by the SUT to the SMTP mail sink.

The Benchmark Manager is the program that reads in the RC files and drives the load generators. You can only have one benchmark manager. The benchmark manager is responsible for starting and stopping the load generator and for collecting results and creating the raw results file. The system that runs the benchmark manager is known as the Primary Client.

1.2. Overview

Below is a quick overview of what needs to be done before running the SPECmail2008 benchmark:

  1. Install and setup your Mail Server according to your Mail Server instructions.
  2. On your Mail Server, create (provision) the target number of user accounts.
  3. Optionally, you may have to create target mailboxes associated with each user.
  4. Configure the SUT such that it relays external e-mails to the designated mail sink host and port.
  5. Configure the client machines to drive the SUT. At least two client machines are recommended.
  6. Determine which clients will serve as the benchmark manager, mail sink, and load generators.
  7. Install the SPECmail2008 benchmark manager and load generator software on your clients.
  8. On the primary client, where the benchmark manager resides, modify the SPECimap_sysinfo.rc file to reflect client/SUT configuration information
  9. On the primary client, modify the SPECimap_config.rc to reflect the SUT host names, port numbers, and number of target users.
  10. Verify that mail can be sent and retrieved by the SUT in accordance to how the SUT is defined in the SPECimap_config.rc file.

Use the following steps to check that the configuration is set up correctly:

  1. Setup and start a short run.
  2. Verify that the mail sink is setup properly, by manually connecting to the mail sink during the short test run. The mail sink should display a banner indicating that it is a SPECimap program.
  3. After the short test has completed, examine the mail logs on the Mail Server to ensure local/remote mail was delivered, and local mail was retrieved.
  4. Run the reporter program on the results file, to verify that statistics and other benchmark related information is being recorded during the test run.
  5. After running this short test, the mailboxes for the users involved in this short test should no longer be used in the compliant test user range. The alternative would be to “clean” these mailboxes.

2. How to install SPECmail2008

2.1. Before you begin – a pre-installation checklist

Here is a checklist of items to review before installing the SPECmail2008 benchmark software.

2.1.1. Network Configuration

It is strongly recommended that the benchmark be run in an isolated network, to avoid external network traffic impacting your measured results. 100base-T Ethernet or faster media and switching are recommended for connecting the clients and server. The connections between a load generating client and the SUT must not use a SUT value of TIME_WAIT less then 60 seconds (see the SPECmail2008 Run and Reporting Rules for more details).

2.1.2. Directory Services

Running directory services is optional, and it is your choice whether to run the directory service on the mail server or on a separate system. In either case, if you are running a directory service it must be reported as part of the SUT. Note that if you are using a directory service, it needs to be populated with the user information before the benchmark can be run (see Creating User Accounts on the Server below).

2.1.3. Mail Domain Names

You must choose two domain names, one for local delivery to the mail server and one for remote delivery to be relayed by the mail server. These domain names need to be entered into your internal domain name service, network information service, distributed name service, or other equivalent service for the mail server. Your mail server needs to be configured to relay remote mail to the SMTP mail sink. The domain names you choose should be unique and isolated from the real world.

2.1.4. Disk Space Allocation

Your mail server will need enough disk space to hold the mail store and mail queue based on the SPECmail2008 user load you intend to generate. You can scale appropriately by using the following formula:

Total storage required = 450MB * number of users + 10% overhead

Note that these disk space values are just estimates, the actual amount of disk space required may vary based on your SUT. The benchmark load will not maintain a steady state. Thus the number of messages and message size will not remain constant during the benchmark run so, you may need additional space on your SUT for disk fragmentation, log files, and storage overhead.

2.1.5. Mailbox Creation

The mailboxes must be created prior to running the benchmark, using the naming convention specified in Creating User Accounts on the Server below. The mailbox contents can be initialized as part of the benchmark run or initialization can be done independently using the –init flag with the benchmark manager.

2.1.6. Client Configuration

The client systems running the benchmark manager and load generators must all use the same version of the Java Runtime Environment (JRE). The load generators require at a minimum JRE v1.5.0. The software has been tested using v1.5.0_1, v1.5.0_6 and v1.5.0_9 on several different platforms. The number of load generators needed depends on the number of SPECmail2008 users being simulated and the performance of the clients –. We recommend that you look at existing posted results for an indication of the number and size of clients you will need to run the benchmark. There is no requirement that all clients be the same type or size.

Note: Using multi-CPU/core hardware with more than four (4) cores (4x1, 2x2 or 1x4)
The best way to utilized the full computational capacity is to run multiple load generators instances on each box, not one load generator. Each load generator spins off one thread per IMAP connection. Java VM's do better when managing a smaller number of concurrent threads. The configuration files and executables support an optional command line parameter (-p portNumber) that causes it to listen on designated port (portNumber) instead of the default.

2.1.7. Synchronizing the Clocks of the Servers and Clients

Synchronization of the clocks on the client and server is not required for the SPECmail2008 benchmark, but it is recommended. Synchronized clocks will simplify evaluating the log files created by the benchmark on the clients.

2.2. Setting Up the Server

2.2.1. Installing your Server

Please see the installation instructions for your version of the mail server. Your installation must present the appearance and behavior of a single logical server for each protocol to all the mail clients (see the SPECmail2008 Run and Reporting Rules for more information).

You will need to set up local and remote domains as defined in the configuration file. The local domain is for messages stored by your mail server, the remote domain is for messages forwarded to the mail sink (a SPECmail2008 load generator acts as the mail sink during the benchmark run). Note that for some mail servers, both the local and remote domains must resolve (they must be recognized by your system). If the remote domain does not resolve, the server will not forward remote mail to the mail sink.

2.2.2. Testing your Server

Once you have installed and started your mail server, you should perform the following tests on all clients to ensure that the clients can communicate with the mail server. Note that you must set up your domains and at least one user account before attempting these tests:

For SMTP (change the words in italics according to your setup):

# telnet <your-server> 25
<the server should greet you with its server name and version number, etc>
helo domain.com
mail from: test001@domain.com
rcpt to: test002@domain.com
data
Subject: test message for SPECmail2008
 
This is a test for SPECmail2008
.
quit

Then go to your mail server queue repository to check if the mail is queued, or check the mailbox to see if it has been delivered. You can also use the following IMAP4 commands to check if the mail has been delivered:

# telnet <your-server> 143
<the server should greet you with its server name and version number, etc>
a login test002 password
b select Inbox
c fetch 1:* (BODY[])
d logout

2.2.3. Creating User Accounts on the Server

The user accounts you create on your mail server for the benchmark must be of the form <USERNAME_PREFIX><USER_ID>, for example test5 or test1000. The username prefix is defined in the configuration file (see Configuration Parameters below). Each user has a unique user ID, which is a number whose lower and upper bounds are defined in the configuration file.

You can create as many user accounts on the server as you wish; the benchmark will only use the user accounts that are dictated by the username prefix and user IDs specified in the configuration file. It is recommended you create your target maximum number of user accounts. The benchmark does not maintain the contents of the mailboxes in a steady state, so re-initialization may be necessary from run to run. An alternative is to initialize the SUT and then backup data for later restores to the initialized state.

2.3. Setting Up the Clients

2.3.1. Installing Java

The load generators are Java applications, so your clients will need a Java version v1.5 or later. You can obtain Java for your client platform from one of the following web sites:

Refer to the vendor web sites for installing and configuring Java on your clients. Java is not necessary on the mail server unless required by your mail server software.

Note that some versions of Java (such as freeBSD) do not contain a Just In Time Java bytecode compiler (JIT). A JIT compiler significantly reduces the CPU consumption of Java. As a result, depending on the speed of the CPU, without a JIT several clients will be required to generate a load for a relatively small number of users. Since a JIT can yield a 10 to 20-fold speedup over the Java interpreter, you may find that running without a JIT is not practical for producing a SPECmail2008 load.

2.3.2. Installing the SPECmail2008 Software

You need these five files to run the SPECmail2008 benchmark:

The specimap.jar and check.jar files must be present on all the client systems that will be running load generators. The RC files should be installed on the primary client (the system running the benchmark manager) along with the specimap.jar and check.jar files. The full installation includes additional files, including sample scripts (in the “sample_scripts” sub-directory) for running the benchmark manager and the load generators.

Note: Running multiple load generators on same host
You do not need to install multiple copies of the SPECmail2008 software on each load generator host. If you plan to run multiple instances of the SPECmail2008 load client. Just modify the enclosed scripts to start each instance with the command line parameter (-p portNumber). Make sure that each portNumber is unique for that host. The portNumber value does not have to be unique across hosts.

2.4. RC Files

2.4.1. Configuration Parameters

Parameters for identifying clients and the mail server are contained in the SPECimap_config.rc file. This is where your client host names and port numbers are listed for the benchmark manager to use when controlling the load generators. This is also where you identify the SMTP server, the IMAP4 server, and the mail sink server name and port number.

At a minimum, to run a test you will need to modify the following parameters:

See Appendix A for a detailed description of the configuration parameters.

Note: Running multiple load generators on same host
The CLIENTS configuration key must list each load generator instance separately. If your host's name is zoog and it runs four (4) load generator instances starting on port 1200, then you should use zoog:1201,zoog:1202,zoog:1203,zoog:1204[others...] in the list of load generator hosts.

2.4.2. System Under Test (SUT) Description

The SUT is defined in SPECimap_sysinfo.rc file. This information is used to create a section describing the SUT in the HTML results file. This information is not necessary for running the benchmark, but it is necessary if you intend to publish results. The SPECmail2008 Run and Reporting Rules specifies that a published result should contain enough information about the SUT to allow someone else to reproduce the results.

Since a SUT may consist of more than one system, most of the parameters use an array format to allow you to identify multiple systems. For example, the function definition for the first system in your configuration uses the parameter SYSTEM.FUNCTION[0]. Additional systems would use the parameters SYSTEM.FUNCTION[1], SYSTEM.FUNCTION[2], SYSTEM.FUNCTION[3], etc. The main exceptions are SYSTEM.TITLE, which is used to name the SUT, and SYSTEM.CONFIG_DIAG, which is used to indicate a file containing a diagram of the SUT.

A description of the clients running the load generators is not required, but is recommended. The clients must appear in the configuration diagram, so placing the hardware and software descriptions in the SUT description will help when evaluating the results.

See Appendix B for detailed descriptions of the SUT parameters.

2.4.3. Trace Parameters

The SPECimap_config.rc file contains a section with trace variables. These trace variables are commented out initially, if you uncomment one or more of them the clients will produce trace information while running the benchmark. This trace information is provided for testing and debugging purposes. You can use the traces when initially setting up your mail server or to track down problems if you are having difficulty with your configuration.

As an alternative, you can disable the trace variables by uncomment them and changing their value to zero. Then to enable them you simply change their value to one.

See Appendix C for a detailed description of the trace parameters.

These trace variables will have a performance impact on the clients, so they should be commented out when running the benchmark to produce publishable results.

2.4.4. Fixed Parameters

The SPECimap_fixed.rc file contains the parameters used by the benchmark to create the compliant workload generated for the mail server, establish run times, and measure the mail server Quality of Service (QoS) criteria. You should not edit this file. If you would like to alter one of these parameters while testing your mail server, you can override these values by putting them in the SPECimap_config.rc file. Any of these values found in the SPECimap_config.rc file will override the corresponding value in the SPECimap_fixed.rc file, and a note will be entered into the results indicating that a fixed parameter was overridden. You cannot produce publishable results if any of these parameters are altered.

See Appendix D for a detailed description of the fixed parameters.

When running the benchmark, the benchmark manager checks the SPECimap_fixed.rc, if it is altered in any way the run will be marked non-compliant.

3. How to Run SPECmail2008

3.1. Starting the Load Generators

The benchmark manager controls the benchmark load generators, but the user must start them manually. Scripts are provided for starting the load generators, but you will need to modify them to work in your environment.

3.1.1. Starting the Load Generators on UNIX

If you are using Java 1.5 or later and java is in your path, the simplest way to launch the client would be to use the command:

  # java -classpath specimap.jar:check.jar specimapclient [-p portNumber]

The default client port is 1099.

You can check which version of java your are using by issuing the following command:

  # java -version

As an alternative, you can put the jar files in your CLASSPATH environment variable, using one of the following commands:

In Bourne/Korn Shell:

  # CLASSPATH=<install dir>/specimap.jar:<install dir>/check.jar
  # export CLASSPATH

In C-Shell:

  # setenv CLASSPATH <install dir>/specimap.jar:<install dir>/check.jar

Then you can run the client using:

  # java specimapclient

Note that the specimap.jar file must be the first file in your classpath.

The default heap size for your Java VM may be insufficient; setting the memory heap size to 256Mb (via the –mx256m flag) or larger should provide enough memory for running the load generator. If your client aborts with memory exceptions, try increasing this value.

The memory heap size must be scaled to match the number of target users.  The benchmark collects and holds runtime statistics in memory as the test progresses.  Therefore, memory heap sizes that work with 200 users will be insufficient for 2000 users.  Some experimentation will be required to tune this value for your environment (number of load generators and number of users).

The load generators have a single flag, -p <port>, which allows you to set the port number the specimapclient will listen on for the benchmark manager. The port number given on the command line must match the port number indicated for that client in the configuration parameters file. If the port is not provided, a default port number of 1099 is used. You will need to alter the port number if you are running multiple instances of the load generator on the same client machine or have a port number conflict with other software running on the system.  Port numbers below 1024 are generally reserved and should not be used for specimapclient.

Thus, an extended version of the command used to start the load generator using a different port number would look like this, assuming your CLASSPATH variable has been set:

  # java –mx256m specimapclient -p 1555

Shell scripts for starting the client are provided with the benchmark; you can modify and use them as well.

3.1.2. Starting the Load Generators on MS Windows

To run the load generators, it is simplest if the specimap.jar and check.jar files are in your CLASSPATH environment variable. To add the jar files to your CLASSPATH in MS Windows, do the following (this example is for MS Windows 2000, your system might differ slightly):

Note that the specimap.jar file must be the first file in your CLASSPATH. 

Once CLASSPATH is set, open a command window and issue the command:

  C:\> java specimapclient

The default heap size for your Java VM may be insufficient; setting the memory heap size to 256Mb (via the –mx256m flag) should provide enough memory for running the load generator (assuming that your client has 512Mb of RAM). If your client aborts with memory exceptions, try increasing this value.

The memory heap size must be scaled to match the number of target users.  The benchmark collects and holds runtime statistics in memory as the test progresses.  Therefore, memory heap sizes that work with 200 users will be insufficient for 2000 users.  Some experimentation will be required to tune this value for your environment (number of load generators and number of users).

The load generators have a single flag, -p <port>, which allows you to set the port number the specimapclient will listen on for the benchmark manager. When starting on a client, the port number given on the command line must match the port number indicated for that client in the configuration parameters file. If the port is not provided, a default port number of 1099 is used. You will need to alter the port number if you are running multiple instances of the load generator on the same client machine or have a port number conflict with other software running on the system.

Thus, an extended version of the command used to start the load generator using a different port would look like this:

  C:\> java –mx256m specimapclient -p 1555

Another way to start the load generator on MS Windows is to use a batch script. Batch scripts for starting the client are provided with the benchmark; you can modify and use them.

3.2. Starting the Mail Sink

The mail sink is started the same way you start the load generators; please see the section above. The specimapclient is used in both cases. A mail sink can also be a load generator, depending on how the configuration file is set up. Note that at this time only one system can act as a mail sink.

You will not be able to verify that the mail sink id started correctly until the benchmark begins the warm-up period. Just prior to the warm-up period, the Benchmark Manager connects to and starts up the mail sink. After the warm-up period has started, you can verify the mail sink has started correctly by performing the following:

  telnet <mail sink hostname> <SINK_PORT>

The SINK_PORT is the port number the mail sink will listen on for receiving outbound messages from the benchmark. A specimapclient banner will be displayed after connecting to the mail sink. Typically, mail servers use port 25 to relay outbound messages. Therefore, if your mail server is configured to use port 25 as the sink port, ensure there are no other processes using port 25 (e.g. the sendmail process).

3.3. Starting the Benchmark Manager

Starting the benchmark manager is similar to starting the load generators; the main difference is that you use specimap rather than specimapclient in the java command and provide command line flags to control the behavior of the benchmark. Scripts are provided for running the benchmark manager in both UNIX and MS Windows; you can also refer to them for examples of how to run the benchmark manager in your environment.

3.3.1. Options

Several options are available when launching the benchmark manager.

Supported Command Line Parameters
Parameter Benchmark Mode/Actions
-v[ersion] Prints the benchmark version number and exits.
-u[sage] Prints the list of command line options, usage information and exits.
-init This command line flag instructs the benchmark manager to initialize the mailboxes on the mail server and then terminate. The manager first empties all the mailboxes in the user range and then adds messages to the mail boxes based on the benchmark’s defined steady state.
-clean This command line flag instructs the benchmark manager to delete all messages and folders in the mailboxes.
-verify This command line flag instructs the benchmark manager to verify the mailboxes are in the compliance with the expected folder structure, message quantity and message size distributions. When running the benchmark, prior to starting a measurement run, the mailboxes are checked to ensure that the message store has the right distributions. This command line flag runs the verify step, indicates the result and terminates so you can check the state of your server prior to performing a compliant run.
-loadtest This command line flag instructs the benchmark manager to initiate the peak hour work load.  Note: This is not the same as a compliant run, even though it generates the same work load.
-compliant This command line flag instructs the benchmark manager to check the RC files to ensure that the ensuing run will be compliant (i.e. no fixed parameters have been overridden). If the run will not be compliant, the benchmark manager terminates with warnings. Without this flag, the manager will still run the benchmark with non-complaint settings and will mark the results as invalid.
-config <filename> This command line flag instructs the benchmark manager to use the identified file for the configuration parameters instead of the default Specimap_config.rc file. This flag is provided so you can use different configuration files for testing and/or place the configuration file in a different directory.
-fixed <filename> This command line flag instructs the benchmark manager to use the identified file for the fixed parameters instead of the default Specimap_fixed.rc file. This flag is provided so you can place the fixed parameter file in a different directory or use a different file when testing.

So to initialize a new mail server (after setting up the user accounts) and running the benchmark manager on UNIX, use the following command:

  # java specimap -init

In MS Windows:

  C:\> java specimap -init

To check and see if the new mail server has been properly initialized or can pass the various folder and message distribution verification, use the following command:

  # java specimap -verify

In MS Windows:

  C:\> java specimap -verify

To start a compliant run on UNIX, use the following command:

  # java specimap -compliant

In MS Windows:

  C:\> java specimap -complaint

To remove all messages and/or folders for all configured user mailboxes on the mail server, use the following command:

  # java specimap -clean

In MS Windows:

  C:\> java specimap -clean

4. Results

After completing a run, the benchmark manager will create a directory named result-yyyyMMddhhmm and generate an output.raw file containing the raw results from the benchmark run. The directory name “result- yyyyMMddhhmm” indicate the date and time the benchmark was started. SPECmail2008 includes a utility for generating an HTML results file called the reporter. The reporter is written in Java, so running it is similar to starting the load generators or benchmark manager.

4.1. Generating a SPECmail2008 HTML Report

4.1.1. In UNIX

To run the reporter, it is simplest if the specimap.jar file is in your classpath. To add the specimap.jar file in your CLASSPATH, use one of the following commands:

In Bourne/Korn Shell:

  # CLASSPATH=<install dir>/specimap.jar:<install dir>/check.jar
  # export CLASSPATH

In C-Shell:

  # setenv CLASSPATH <install dir>/specimap.jar

Then cd into the directory where the output.raw file is. You run the reporter using:

  # java specimapreporter <filename>.raw

Where <filename>.raw is the raw file generated by the benchmark. By default this file will be called output.raw. The reporter will generate an HTML file of the form <filename>.html. The reporter also supports a ‘-d’ flag for producing a more detailed report. This report contains additional information you can use when evaluating the performance of your mail server. To generate a detailed report for output.raw, use the command:

  # java specimapreporter -d output.raw

4.1.2. In MS Windows

To run the reporter, it is simplest if the specimap.jar file is in your classpath. To add the specimap.jar file in your CLASSPATH, do the following (this example is for MS Windows 2000, your system might differ slightly):

Then, open a command (“DOS”) window and cd into the directory where the output.raw file is. You run the reporter using:

  C:\> java specimapreporter <filename>.raw

Where <filename>.raw is the raw file generated by the benchmark. By default this file will be called output.raw. The reporter will generate an HTML file of the form <filename>.html. The reporter also supports a ‘-d’ flag for producing a more detailed report. This report contains additional information you can use when evaluating the performance of your mail server. To generate a detailed report for output.raw, use the command:

 C:\> java specimapreporter -d output.raw

4.2. The HTML Results File

The HTML file created by the reporter is designed to be consistent with other SPEC reporting pages. It contains the metric of the benchmark run, a summary of the key results, and a complete system description. The detailed report includes additional information about the benchmark at the three different load levels.

4.2.1. Results Block

The results block contains the key information for the benchmark results. It includes the system name and results. It also includes publishing information such as who produced the results and when the results were created. In the case of an invalid run, a description below the results block will contain a link to the Error section that contains detailed information about why the run did not produce a valid (i.e. publishable) result.

4.2.2. Summary Results

The summary results section contains the Quality of Service (QoS) performance of the mail server on the key functions. These functions are the primary items of concern when testing a mail server using the SPECmail benchmark, and thus the strengths and weaknesses of a mail server can be determined from this table. It includes the following items:

Key Function

Criteria

SMTP Connect

QoS measure for how well the mail server responds to a SMTP connection request

SMTP Data

QoS measure for how well the mail server responds to a SMTP data command (which contains the body of an outgoing mail message)

IMAP Connect

QoS measure for how well the mail server responds to an IMAP4 connection request

IMAP Select

QoS measure for how well the mail server responds to an IMAP4 folder select request, which is how a mail user requests information on new mail in his mailbox

IMAP List

QoS measure for how well the mail server responds to an IMAP4 folder list request.

IMAP Fetch Uid BodyAll

QoS measure for how well the mail server responds to an IMAP4 retrieve-by-Message_UID request, which is how a mail user extracts messages from the server

IMAP Fetch SeqNum Body

QoS measure for how well the mail server responds to an IMAP4 retrieve-by-Message_sequence_number request, which is how a mail user extracts messages from the server

IMAP Store Set Flags

QoS measure for how well the mail server responds to an IMAP4 message status flag change request

Error Rate

QoS measure for how well the mail server responds under load

4.2.3. System Descriptions

The system description section contains detailed information about the SUT. This information is extracted from the configuration file, which the benchmarker sets up prior to running the benchmark. It is intended to provide information for viewers to use when evaluating different systems and comparing results. The SPECmail2008 Run and Reporting Rules specify that enough information be provided so that a third party could set up a system and duplicate the results published.

4.2.4. Configuration Diagram

When submitting results to SPEC for publication, you must include an image file with a drawing of you mail server configuration. The HTML file created by the reporter will contain a link to this file using the file name specific in the configuration file. The file must at a minimum indicate the network connections between the mail server and the load generators. The image must be in GIF, JPEG, or PNG format.

4.2.5. Detailed Results

If the ‘-d’ flag is used when creating the result file, a detailed results section for each load is included in the HTML file. This detail section includes more information on the summary results (above) and additional performance details recorded by the load generators. The detail section includes:

4.2.6. Notes/Tuning Information

The Notes/Tuning Information section contains additional information on the SUT configuration and tuning parameters.

If the results failed validation for some reason, an Errors block will be added to the end of the Notes/Tuning Information section with a detailed description of what caused the failure.

5. Tips and Troubleshooting

Tips and Troubleshooting contains additional information on running the benchmark.

5.1. Using Java Version v1.5

The SPEC Mail Server subcommittee suggests you use Java version 1.5 or later. 

5.2. Benchmark Manager Only Prints Version Number

The scripts provided to run the benchmark manager use the '-v' flag which prints out the version number of the code.  If you would like to run the benchmark manager using a script, you should modify the provided script to use the appropriate options for your configuration. The first time you run the benchmark, you should use the command:

java specimap -init

which will only initialize the mail servers mailboxes.  This initialization takes a significant amount of time, so by default the benchmark assumes the mailboxes have been initialized prior to starting. The benchmark maintains the mailboxes in a steady state during the run, so under normal conditions you will only need to initialize the mailboxes once. You can verify that the mailboxes were initialized correctly using the command:

java specimap -verify

When performing a benchmark run to generate results for publication, you should use the command:

java specimap -compliant

The -compliant flag will cause the benchmark to terminate if an error or warning is encountered during the run.  Without the -compliant flag, warnings will be printed to the console but the benchmark will continue to run.

5.3. Connecting to Remote Clients Failed: InvalidClassException

When launching the benchmark manager and adding hosts, you may see an error containing text similar to:

Error: Name lookup for load generator alt100 exception error
unmarshalling return; nested exception is:
        java.io.InvalidClassException: specimapclient_Stub; Local 
class not compatible: Stream classdesc serialVersionUID=xxx local
class serial VersionUID=yyy
...

This error indicates that you have non-compatible versions of the benchmark code on your clients.  All clients (running the benchmark manager, load generators and/or mail sink) must have the same version of the benchmark software installed and running in order to run the benchmark.

6. Submitting Results

Once you have a successful run, you can submit the results to the SPEC mail server subcommittee for review by mailing the output.raw file and the configuration diagram to submail2008@spec.org. When mailing the files, include them as attachments to the mail message and only submit one result per email message.

Note: The raw file contains the configuration information from the SPECimap_config.rc file. Please edit the RC file with the correct SUT information prior to running the benchmark for submission.

Every submission goes through a two-week review process, starting on a scheduled Mailserver sub-committee conference call. During the review, members of the committee may ask for additional information or clarification of the submission. Once the result has been reviewed and approved by the committee, it is displayed on the SPEC web site at http://www.spec.org/.

Appendix A – Configuration Parameters

Parameter Name

Usage

CLIENTS

A list of clients that will be running the load generators for the benchmark. A client entry consists of the machine name and the port number the load generator the benchmark manager will use to connect to the client.

SMTP_SERVER
SMTP_PORT

The host name and port number of the SMTP server for your mail server.

IMAP_SERVER
IMAP_PORT

The host name and port number of the IMAP4 server for your mail server.

SINK_SERVER
SINK_PORT

The host name and the port number of the system that will provide mail sink services during the benchmark. Note that mail sink services are provided by a Specimap load generator, thus this system must have a load generator running and using this port.

LOCAL_DOMAIN

The domain name that represents local mail on the mail server (i.e. mail intended for local recipients that must be stored).

REMOTE_DOMAIN

The domain name that represents remote mail on the mail server (i.e. mail not intended for local recipients that must be forwarded).

USERNAME_PREFIX

The prefix to use on the user account names for the mailboxes; user account names are of the form <USERNAME_PREFIX><USER_ID>.

USERNAME_AS_PASSWORD

A flag to indicate that the user account name should also be used as the password (0=no, 1=yes).

USER_PASSWORD

If the account name is not used as the account password, this value will be used as the password for all the user accounts.

USER_START
USER_END

The starting and ending numbers to use for the user account names for the mailboxes; user account names are of the form <USERNAME_PREFIX><USER_ID>.

SMTP_SEND_DELAY_MILLISECONDS

When sending messages to the mailboxes during mailbox initialization, this value indicates how long to wait between messages.

SMTP_INIT_REST_MINUTES

After completing mailbox initialization, the benchmark will ‘rest’ for the specified amount of time to allow the mail server to de-queue all the new messages.

RSL_FILENAME

The RSL file is an ASCII file generated by the benchmark manager when the run is completed that summarizes the raw file data for the run.

Appendix B – System Description Parameters

The following parameters are singletons, they should appear only once in the configuration file.

Parameter Name

Usage

PREPARED_BY

The name of the person that ran and submitted this result.

TESTED.BY

The name of the company that ran and submitted this result.

LICENSE.NUM

If the company is a SPEC member, this is the SPEC license number assigned to that company. If the company is not a SPEC member, this is the license number assigned when the SPECmail2008 license was purchased from SPEC.

TEST.DATE

Month and Year of the benchmark run.
This field must be of the form Mon-YYYY (i.e. “Nov-2000”).

NOTES

General notes to appear in the result file. Multiple notes can be entered, so the parameter uses an array notation. For example:

NOTES[0] = “This is the first notes line.”
NOTES[1] = “”
NOTES[2] = “You can even include blank lines.”

SYSTEM.TITLE

The overall name for the SUT. This is used as the title for your results page.

SYSTEM.CONFIG_DIAG

Name of the file containing your configuration diagram.

The following parameters are system description parameters; they are repeated for each system being described as part of the SUT. The files uses an array context for these parameters, see the provided Specimap_config.rc file for examples.

Parameter Name

Usage

SYSTEM.FUNCTION

The label that will appear at the very top of the system description box.

SYSTEM.NUM_SYSTEMS

The number of systems of this type used in the test.

SYSTEM.HW_MODEL

This is the hardware model of the system.
Note: this field is a searchable field, so model naming should be consistent with other SPEC submissions when publishing results.

SYSTEM.HW_VENDOR

This is the hardware vendor of the system.
Note: this field is a searchable field, so vendor naming should be consistent with other SPEC submissions when publishing results.

SYSTEM.HW_AVAIL

The date the hardware is/will be shipping and generally available to the public.
This field must be of the form Mon-YYYY (i.e. “Nov-2000”).

SYSTEM.OS

Operating System (including version number)

SYSTEM.FILESYSTEM

The file system used.

SYSTEM.SW_LABEL

The label that appears in the Software section of the system description box. A special value of “JVM” entered into this field causes the SW_NAME to be ignored and the JVM and JIT fields to be used instead (for use with client systems running the load generators).

SYSTEM.SW_NAME

The name and version number of the software under test (this field is ignored on client systems).

SYSTEM.SW_AVAIL

The date the software is/will-be shipping and generally available to the public.
This field must be of the form Mon-YYYY (i.e. “Nov-2000”).

SYSTEM.JVM

Java Virtual Machine description and version number. This field is only used if the SYSTEM.SW_LABEL parameter contains “JVM”.

SYSTEM.JIT

Just In Time Java compiler description and version number. This field is only used if the SYSTEM.SW_LABEL parameter contains “JVM”.

SYSTEM.CPU

The type of central processing unit(s) in the system.

SYSTEM.CPUMHZ

The speed (in MHz) of the CPUs.

SYSTEM.CPU_ENABLED

The number of CPUs in the system.

SYSTEM.MEMORY

The amount of physical memory (in Megabytes) in the system. This field should be an integer (do not use “Mb” or “Gb” in this field).

SYSTEM.L1CACHE

The amount of level 1 cache, for instruction (I) and data (D) on each CPU.

SYSTEM.L2CACHE

The amount of level 2 cache on each CPU

SYSTEM.L3CACHE

The amount of level 3 cache on each CPU. If caches higher than level 3 exist, this value should represent all level 3 cache and higher.

SYSTEM.DISK

Size and Type of disks used by the system. If a complex disk system is used, additional information on the disks should be included in the SYSTEM.NOTES parameters.

SYSTEM.NETWORK

Type of network interface(s) used by the system.

SYSTEM.HW_OTHER

Other hardware in the system that is performance-related.

SYSTEM.NOTES

Additional notes about the system.

Appendix C – Trace Parameters

The trace parameters allow you to have the load generators print information messages to standard out when running the benchmark. If you are going to be activating a trace, you should redirect the output from the load generators to a file when you start them on the clients.

These trace variables will have a performance impact on the clients, so they should be commented out when running the benchmark to produce publishable results.

Parameter Name

Usage

TRACE_EVENTS

Print message for all events in the load generators.

TRACE_FOLDERVRFY

Print message for every IMAP4 folder verify event – a check of all folders (mailboxes) in each user’s mail store complies with the expected sub-folder and count distributions.

TRACE_MESSAGEVRFY

Print message for every IMAP4 message verify event – a check of all messages in each user’s mailbox/folder to ensure that the expected size and count distributions are met.

TRACE_IMAPCLEAN

Print message for IMAP4 clean events. Clean events occur when the benchmark empties the mail boxes.

TRACE_FOLDERINIT 
TRACE_MESSAGEINIT

Print message for IMAP4 initialization events specific to creating the target folder structures and populating these folders. Initialization events occur when the mailboxes are pre-populated with the benchmark mail distribution.

TRACE_TP1
TRACE_TP2
TRACE_TP3
TRACE_CS6

Print message for each IMAP4 command associated with specific Traffic Patterns.  Only TP1 through TP5 are used for a compliant run.  However, CS6 is used as a traffic pattern that has no random parameters.

TRACE_LOADGENERATOR

Print message for communications with load generators and their responses.

 

 

TRACE_SMTPQOS

Print message for SMTP QoS events. QoS events are generated when the load generators create, send and retrieve mail message to verify the mail server meets the delivery time criteria.

TRACE_SMTP

Print message for SMTP events.

TRACE_SMTPSINK

Print message for SMTP SINK events. SINK events occur when a load generator, acting as the mail sink, receives message relayed by the mail server that are addressed to remote users.

TRACE_MISC

Print message for miscellaneous events. This prints messages for activity that does not fit well into any other category.

TRACE_MDIM_KEYS

When tracing a vector array, this trace will print out the key values being searched.

TRACE_CLIENT_MSGS

When processing the client list from the configuration file, this trace will print out the clients being extracted from the list.

TRACE_SETMSGSIZES

When processing the message size list from the file, this trace will print out the message sizes being extracted from the list.

TRACE_SETMBOXSIZES

When processing the mailbox size distribution list from the file, this trace will print out the items being extracted from the list.

TRACE_CONFIGREAD

When processing items in the configuration file, this trace will print out the items as they are parsed.

TRACE_READFROM

When processing the parameter files, this trace prints a message when each section is complete.

TRACE_SETRECIP_MSGS

When processing the mail recipient distribution list from the file, this trace will print out the items being extracted form the list.

TRACE_MODEM_RATES

When processing the modem rates to use in the load generators, this trace will print out the rates being extracted form the list.

TRACE_CALCRESULT

When running the reporter to generate HTML results, this trace will print out the values being used and the calculations that are performed.

TRACE_RESULTS

Print message when the load generators record results.

TRACE_MSGSIZE

Print message when the load generators create a new message to send.

TRACE_WORKLOAD

Print message when adding or changing the workloads on the clients.

TRACE_VERIFY_MBOX_COUNT

When verifying the number of messages in the mailboxes matches the expected distribution, this trace will print out the actual and expected message count distribution in the mailboxes.

TRACE_VERIFY_MSG_SIZE

When verifying the size of the messages in the mailboxes matches the expected distribution, this trace will print out the actual and expected message size distribution in the mailboxes.

Appendix D – Fixed Parameters

The fixed parameters are used by the benchmark to setup the load generators. You should never edit this file. If you would like to alter one of these parameters while testing your mail server, you can override these values by putting them in the SPECimap_config.rc file. Any of these values found in the SPECimap_config.rc file will be used in place of the value located in the SPECimap_fixed.rc file, and a note will be entered into the results indicating that a fixed parameter was overridden.

Parameter Name

Usage

WARMUP_SECONDS

How long to run the benchmark at load before starting to gather statistics.

RUN_SECONDS

How long to run the benchmark at load while gathering statistics.

RAMPDOWN_SECONDS

How long to wait after stopping the gathering of statistics and terminating the load before starting a new test (a.k.a. cool down period)

LOAD_FACTORS

The load factors with which we scale the load in a set of consecutive test runs.

MSG_SIZE_DISTRIBUTION

The message size distribution for mail messages sent by the load generators.

MSG_RECP_DISTRIBUTION

The number of recipient’s distribution for mail messages sent by the load generators.

MIME_PART_SIZE_DISTRIBUTION

The probability distribution of the various MIME parts that make up a sing message.

MIME_TOP_PART_DISTRIBUTION

The probability that a MIME level has the sublevels indicated in the various distribution buckets.

MIME_PART_LEVEL

The probability that a message has the number of nested MIME levels indicated in the distribution buckets.

MIME_TOP_PART_COUNT

The probability distribution of the number of MIME parts at the top level

POP_RETRY_BOXES_PERCENT

The percentage of mailboxes set aside for POP retries.

DATA_RATE_DISTRIBUTION

Ignored for this benchmark

SLOW_MODEM_DATA_RATE
FAST_MODEM_DATA_RATE
CABLE_MODEM_DATA_RATE

Characters per second for 28.8Kbps, 56.6Kbps and cable modems.  These are ignored in the IMAP benchmark.

PROTOCOL_TIMEOUT_ SECONDS

The number of seconds after which a connection is deemed to have timed out.

PROTOCOL_TIMEOUT_ PERCENT

The percentage of connections that should NOT time out.

DISCONNECT_PERCENT

The percentage of client connections that will unexpectedly disconnect (i.e. the load generators will not properly terminate the connection).

REMOTE_REFUSED_PERCENT

Percent of remote deliveries will will be refused immediatley, forcing local queueing.

REMOTE_DELAY_DISTRIBUTION

Distribution of response delays (in milliseconds) for accepted SMTP connections (versus REFUSED).

DELIVERY_TIME_CHECK_PERCENT

The percentage of local SMTP messages that should be check for delivery time Quality of Service (QoS).

SMTP_DELIVERY_QOS_SECONDS

The number of seconds within which a local SMTP message should be delivered to meet the delivery time QoS limit.

SMTP_DELIVERY_QOS_PERCENT

The percentage of local SMTP messages that need to meet the delivery time QoS limit at 100% load in a valid benchmark run.

SIMPLE_COMMAND_QOS_SECONDS

The number of seconds within which the mail server should respond to a simple mail command.

SIMPLE_COMMAND_QOS_PERCENT

The percentage of simple mail commands that need to pass the QoS requirement at 100% load in a valid benchmark run.

COMPLEX_COMMAND_QOS_SECONDS

The number of seconds within which the mail server should respond to a complex mail command.

COMPLEX_COMMAND_QOS_PERCENT

The percentage of complex mail commands that need to pass the QoS requirement at 100% load in a valid benchmark run.

MSG_DESTINATION_LOCAL_PERCENT

The percentage of the mail generated by local users that is destined for other local users. The rest of the locally originated mail is destined for remote users, and all mail originating remotely is destined for local users. See the White Paper for more information.

PEAK_PCT_USERS

Percent of provisioned users receiving messages during the peak hour (aka active users)

MSG_RECEIVED_PER_PEAK_HOUR

Number of messages received by active users during the peak hour

LOCAL_TO_LOCAL_PCT

Percent of total messages sent from local to remote users

REMOTE_TO_LOCAL_PCT

Percent of total messages sent from remote to local users

LOCAL_TO_REMOTE_PCT

Percent of total messages sent from local to remote users

MSG_SENT_PER_DAY

The number (on average) of messages sent by local users every day.

PEAK_LOAD_PERCENT

The percentage of the daily load occurring during the peak hour (this is the load simulated by specimap at 100% load factor).

IA_ADJUSTMENT

Initialization seed for Inter-arrival time computation

CS1_PRIMARY_PCT

Percent of CS1 sessions already logged into the mail server and should already be connected at the beginning of the load test period.

CS2_PRIMARY_PCT

Percent of CS3 sessions already logged into the mail server and should already be connected at the beginning of the load test period.

CLIENT_TYPE_DISTRIBUTION

Distribution of load generator threads that will implement the specific combination of command sequences associated with each client type found in the sample data

LEVELXFOLDER[arrayIndex]

Distribution defining probability of number of subfolders at each level.  The arrayIndex value indicates the folder depth (nesting level).

LEVELXWITHSUB[arrayIndex]

Distribution defining probability that each folder has zero or more subfolders at that depth.  The arrayIndex value indicates the actual folder depth (nesting level).

USE_PERCENT_FOR_VERIFICATION

Percent of active user base to use for folder and message distribution verification.

CS6A
CS6R

The number of messages Appended and messages Retrieved during CS6 fixed load test


Copyright (c) 2008 Standard Performance Evaluation Corporation