SPECweb96 Q&A

Answers to Common Questions About the SPECweb96 Benchmark

---

Q1: What is SPECweb96?

A1:

SPECweb96 is a software benchmark product developed by the Standard Performance Evaluation Corp. (SPEC), a non-profit group of computer vendors, systems integrators, universities, research organizations, publishers and consultants. It is designed to measure a system's ability to act as a World Wide Web server for static pages.

SPECweb96 is targeted at systems designed to handle millions of hits per day and multiple hits per second. It is the first step in giving Web users the most objective, most representative benchmarks for measuring Web server performance. Subsequent releases will address such issues as encryption, CGI and multimedia.

Q2: What organizations were involved in the development of SPECweb96?

A2:

SPECweb96 benefited from the development efforts of the following organizations: CommerceNet, Digital Equipment Corp., HAL Computer Systems, Hewlett-Packard, IBM, Intel, Netscape, OpenMarket, Siemens Nixdorf Informationssysteme, Silicon Graphics, Spyglass, and Sun Microsystems.

Q3: When and where will SPECweb96 results be available?

A3:

Initial SPECweb96 results are available on SPEC's Web site: http://www.spec.org/osg/web96/. Subsequent results will be posted on an ongoing basis following SPEC's two-week review cycle.

Q4: What is the difference between SPECweb96 and other Web server benchmarks such as WebStone?

A4:

Unlike previous Web server benchmarks, SPECweb96 is a standardized test. The SPEC membership -- leading vendors, systems integrators, universities, research organizations, publishers and consultants -- has agreed on a single benchmark with one standardized implementation and workload.

Before any SPECweb96 results are published, SPEC requires that the system tested and the workload measured adhere to agreed-upon standards. All results available through SPEC include full disclosure information that reveals exactly what configurations have been used to obtain a particular result.

The ability to compare results is a key advantage of SPECweb96. Existing Web server benchmarks can be effective tools; in fact, several of them have features not currently available in SPECweb96. But, the plethora of versions and alternative implementations of these benchmarks make it extremely difficult to compare results obtained from different sources.

SPECweb96 also provides a clear advantage to those who cannot run their own Web server performance tests. They can turn to SPEC for standardized, comparable results.

Q5: Can SPECweb96 numbers be compared to WebStone numbers?

A5:

No. Although both benchmarks report the number of HTTP operations per second, their workloads are different. Due to its larger fileset sizes and its pattern of access, SPECweb96 tends to put more stress on a system's memory and its I/O systems. As a result, there is no simple scaling between WebStone and SPECweb96 numbers.

Q6: What kind of workload is used in the SPECweb96 benchmark?

A6:

The SPECweb96 workload simulates the accesses to a Web service provider, where the server supports the home page for a number of different organizations. Each home page is a collection of files ranging in size from small icons to large documents or images. As in the real world, certain files within the home page are more popular than others.

The workload is based on analyses of server logs from a variety of popular Internet servers and some smaller Web sites. To further validate the workload, data from the analyses was compared to logs from Netscape and CommerceNet.

In this initial release, SPECweb focuses on server performance for static Web pages. Dynamic services are developing too rapidly at this point to allow a standardized benchmark to be developed. SPECweb96's workload, therefore, is based on HTTP "get" requests.

The SPECweb96 workload defines four classes of files to get, based on the following file sizes: less than 1 KB, 1 to 10 KB, 10 to 100 KB, and 100 KB to 1 MB. There are several files in each class, with sizes distributed evenly through the range for that class.

The access patterns to the files were determined from the analyses of the Web server logs. SPECweb96 directs 35 percent of its activity to the smallest class, 50 percent to the 1-to-10-KB class, 14 percent to the 10-to-100-KB class, and one percent to the largest files. Within each class there are non-linear distributions of accesses, reflecting the fact that certain files are more popular than others. Finally, the number of times each class set is serviced is determined by the amount of throughput the server can handle -- a high-end server will service a greater number of files than a small desktop server.

Q7: What are the limitations of SPECweb96?

A7:

SPECweb96 is a standardized benchmark, which means that it is an abstraction of the real world. There are several key simplifications that have been made in order to keep SPECweb96 from becoming unmanageable.

Also, SPECweb96 does not attempt to model latency associated with obtaining data across a wide-area network (WAN) such as the Internet. This kind of behavior is difficult to simulate at this stage, since it requires elaborate hardware and software. The reported response times for SPECweb96, therefore, are for the server only; they do not take into account any time delays in getting across the network to and from that server.

Q8: How can I use SPECweb96 to determine the best server for my needs?

A8:

This is somewhat like trying to answer who is the world's best athlete or what is the best car. It all depends on what you are looking for. And, of course, no benchmark will measure important factors such as brand loyalty, a good salesperson, dependability, service and support.

SPECweb96 does define one interesting level of competition, however. SPEC has developed a standard set of definitions for what we think is an interesting test of capabilities. Major Web server hardware and software vendors have agreed that this is a valid test. We expect to see SPECweb96 results from most of the important Web server system vendors.

Although SPECweb96 is not a silver bullet that can provide the entire answer to what server is best for specific needs, it does provide a useful set of information about key parts of the competition. And, like every SPEC benchmark, SPECweb96 will continue to evolve in step with users' needs.

Q9: How can I use SPECweb96 to determine the size of the server I need?

A9:

SPECweb96 is probably not the right tool for sizing a server. SPECweb96 offers data about one interesting workload within a nearly infinite universe of possible workloads. The SPECweb96 workload might or might not represent your workload well.

You can still learn a great deal, however, by looking at how servers handle the SPECweb96 workload. The workload uncovers several key components of a good Web server, including LAN performance, processing power, and memory bandwidth, to name a few. Furthermore, since the workload is standardized, all data points are related to the same units of work.

Q10: Does SPECweb96 support Keep-Alive?

A10:

No. SPEC realizes that Keep-Alive is a very important performance feature, but unfortunately, not enough is known yet about how it affects server behavior. We cannot, therefore, provide a standardized test for Keep-Alive.

Keep-Alive would have much more of an impact on benchmark performance than on real-world performance. In benchmark testing, a relatively small number of the same clients are constantly accessing the same server. Real-world performance is much less constant -- it is common for someone to get a few files, think about it for a while, get one or two more files, go off to visit a Web site, come back for a few more files, and so on.

Within a benchmarking environment, Keep-Alive would allow hundreds of pages to pass through a single connection. SPEC knows that this would not be representative of the real world, but we do not currently have enough experience to properly model Keep-Alive to reflect less-predictable human behavior and dynamically sized files. As a result of not supporting Keep-Alive, SPECweb96 results will tend to be conservative, rather than overstating a server's capabilities.

Q11: What is included with SPECweb96?

A11:

The benchmark comes with the code necessary to build and run the driver, as well as the code for building the dataset that will be requested from the Web server. It is up to the person running the benchmarks to provide the Web server and to install the dataset created by the SPEC tools.

SPECweb96 does not provide Web server software, but any Web server that is HTTP/1.0-compliant can be used. In addition to licensed server software, public domain software such as Apache <http://www.apache.org> can be used to run SPECweb96.

The SPECweb96 CD-ROM contains distributions for Windows NT and Posix-compliant UNIX driver systems.

Q12: What additional software is required to run SPECweb96?

A12:

First, of course, you'll need a properly running Web server. A full ANSI-C programming environment is required to build the benchmark from the provided source code. Binaries will be provided for some system architectures. A Perl5 interpreter is needed to run the benchmark tool environment, which is necessary to obtain results. On Windows NT systems, REXEC protocol support is needed to drive from multiple client systems. SPEC provides this support in SPECweb96 through an implementation based on a public domain package.

Q13: What hardware is required to run the benchmark?

A13:

SPECweb96 is designed to drive the Web server from a number of other client systems. The number of client systems needed will depend on the performance of the Web server, as well as the capabilities of available clients.

The local-area network (LAN) is a very important component of Web server performance. Care should be taken not to overload any part of the LAN, including any hubs, switches, or the LAN segments themselves. The need for an effective LAN configuration might determine the number of client systems required for a high level of performance.

Q14: How can I obtain the benchmark?

A14:

SPECweb96 is scheduled for public release in late August. It will be available on CD-ROM for $800. To order, contact SPEC's administrative office: SPEC, 10754 Ambassador Drive, Ste. 201, Manassas, VA 20109; tel: 703-331-0180; fax: 703-331-0181; e-mail: info@spec.org.