|
Results
Information
Press and Publications
Order Benchmarks
Resources
|
Frequently Asked Questions about HPC2002
Last Updated: 12 Dec 2002
Contents
-
Q1. What is SPEC?
-
SPEC is an acronym for the Standard Performance Evaluation
Corporation. SPEC is a non-profit organization composed of computer
vendors, systems integrators, universities, research organizations,
publishers and consultants whose goal is to establish, maintain and
endorse a standardized set of relevant benchmarks for computer
systems. Although no one set of tests can fully characterize overall
system performance, SPEC believes that the user community will
benefit from an objective series of tests that can serve as a common
reference point.
-
Q2. What is a benchmark?
-
The definition from Webster's II Dictionary states that a
benchmark is "A standard of measurement or evaluation." A
computer benchmark is typically a computer program that performs a
strictly defined set of operations (a workload) and returns some form
of result (a metric) describing how the tested computer performed.
Computer benchmark metrics usually measure speed (how fast was the
workload completed) or throughput (how many workloads per unit time
were measured). Running the same computer benchmark on multiple
computers allows a comparison to be made.
-
Q3. Why use a benchmark?
-
Ideally, the best comparison test for systems would be your own
application with your own workload. Unfortunately, it is often very
difficult to get a wide base of reliable, repeatable and comparable
measurements for different systems on your own application with your
own workload. This might be due to time, money, confidentiality, or
other constraints.
-
Q4. What options are viable in this case?
-
At this point, you can consider using standardized benchmarks as a
reference point. Ideally, a standardized benchmark will be portable
and maybe already run on the platforms that you are interested in.
However, before you consider the results you need to be sure that you
understand the correlation between your application/computing needs
and what the benchmark is measuring. Are the workloads similar and do
they have the same characteristics? Based on your answers to these
questions, you can begin to see how the benchmark might approximate
your reality.
Note: It is not intended that the SPEC benchmark suites be used as a
replacement for the benchmarking of actual customer applications to
determine vendor or product selection.
-
Q5: Does SPEC encourage using its benchmarks for
research?
-
Yes. Although much of the benchmarking documentation is written for a
benchmarker who intends to generate SPEC-publishable results, SPEC
has defined guidelines for research use of its benchmarks (see http://www.spec.org/hpg/academic_rules.html).
These guidelines are intended to be consistent with those for
high-quality scientific work.
-
Q6: I only want to use the SPEC benchmarks for my research. Do
I need to understand the "SPEC Philosophy" ?
-
The SPEC benchmarking philosophy deals with fairness in obtaining and
reporting computer performance measurements. Many of these ideas
apply to scientific performance evaluation as well. Familiarizing
yourself with the ideas behind SPEC benchmarks could prove useful for
your research and the publication quality of your results.
-
Q7: Are any of the SPEC rules binding for me?
-
The only way you could have legally obtained access to the SPEC
benchmarks is by becoming a licensee of SPEC. (Your advisor, or
colleague who gave you the codes might be the licensee, and he or she
has agreed to abide by SPEC's rules). However, most rules apply
to the process of generating SPEC-publishable benchmark results. For
research use, SPEC has defined guidelines (see http://www.spec.org/hpg/academic_rules.html)
-
Q8: Are any of the benchmark run tools useful for my
research?
-
The goal of SPEC's benchmark run tools is to help the
benchmarker, enforce SPEC's runrules, and ensure the quality of
the benchmark reports. Some important aspects of the tools are:
-
Benchmark code and even suites can be made, run, validated and
reports generated with a single command line.
-
The SPEC-provided makefiles are platform-independent (among the
systems whose manufacturers participate in SPEC). System-specific
make commands are separated out into a so-called config file.
-
When making and running a benchmark, the tools copy the source and
all relevant files into a completely separate "run"
directory, isolating the run from the original source and from
other runs.
All these facilities can be useful for research projects as well. It
might be worth learning about the tools.
-
Q9: I want to use the source code and input files only, I
don't want to learn about the run tools. How do I proceed?
-
SPEC HPC benchmarks are full applications, which cannot be made with
a simple "f77 *.f" command and executed with
"a.out<inputfile". Most benchmarks are a mix of Fortran
and C source files. The source files include several parallelization
options (such as OpenMP, MPI, or serial). Invoking a benchmark might
take several runs of one or several executables. Each benchmark must
be validated by comparing its output against a standard file, whereby
certain tolerances are allowed. This process is specified in a file,
Spec/object.pm, in each benchmark directory. Understanding object.pm
requires tool-specific knowledge. The easier option would be to run
the benchmark once using the SPEC run tools and then check the files,
speccmds.cmd and compare.cmd, in the run directory. They contain the
commands for running and validating the benchmarks, respectively.
-
Q10: What is SPEC HPC2002?
-
SPEC HPC2002 is a software benchmark product produced by the Standard
Performance Evaluation Corp.'s High-Performance Group (SPEC/HPG).
SPEC is a non-profit organization that includes computer vendors,
systems integrators, universities, research organizations, publishers
and consultants from around the world. The benchmark is designed to
provide performance measurements that can be used to compare
compute-intensive parallel workloads on different parallel computing
systems.
-
Q11. What does SPEC HPC2002 measure?
-
SPEC HPC2002 focuses on high-performance computer platforms. The
benchmarks measure the overall performance of high-end computer
systems, including
-
the computer's processors (CPUs),
-
the interconnection system (shared or distributed memory),
-
the compilers
-
the MPI and/or OpenMP parallel library implementation
-
the input/output system.
It is important to remember the contribution of all these components;
performance is more than just the processor.
SPEC HPC2002 is made up of several large applications that represent
real computing practices in their respective disciplines.
Note that SPEC HPC2002 does not overly stress other computer
components such as I/O (disk drives), external networking, operating
system or graphics. It is possible to underconfigure a system
in such a way that one or more of the components impact the
performance of HPC2002 (e.g. running a benchmark on a system
with 32 CPU and a single disk for I/O). The HPC2002 benchmarks
are derived from real HPC applications and application practices.
Although it is not the intent of the benchmark suite to stress
certain system components, it is possible to do so on some system
configurations.
-
Q12. What is included in the SPEC HPC2002 package?
-
SPEC provides the following on the SPEC HPC2002 media:
-
Source code and datasets for the HPC2002 benchmarks
-
A tool set for compiling, running, validating and reporting on the
benchmarks
-
Pre-compiled tools for a variety of operating systems and hardware
platforms.
-
Source code for the SPEC HPC2002 tools, for systems not covered by
the pre-compiled tools
-
Run and reporting rules defining how the benchmarks should be used
to produce SPEC HPC2002 results.
-
Documentation
-
Q13: What applications are included with SPEC HPC2002?
-
The SPEC HPC2002 suite includes three application areas: seismic
processing (SPECseis), computational chemistry (SPECchem), and
climate modeling (SPECenv).
SPECseis contains Seismic, an application developed at Atlantic
Richfield Corp. (ARCO). Seismic performs time and depth migrations
used to locate gas and oil deposits.
SPECchem contains GAMESS (General Atomic and Molecular Electronic
Structure System), an improved version of programs that originated in
the Department of Energy's National Resource for Computations in
Chemistry. Many of the functions found in GAMESS are duplicated in
commercial packages used in the pharmaceutical and chemical
industries for drug design and bonding analysis.
SPECenv is WRF, a weather forecast application.
More detailed descriptions of the applications (with reference to
papers, web sites, etc.) can be found in the individual benchmark
directories in the SPEC benchmark distribution.
-
Q14: Are different-sized datasets available for the two SPEC
HPC2002 applications?
-
Yes. Vendors may report SPEC HPC2002 results based on any of two
predefined problem sizes: small and medium. Problem sizes depend on
the application and the type of analysis the application performs.
For SPECseis, problem sizes relate directly to the number and size of
seismic traces being processed. For SPECchem, problem sizes relate to
the complexity of the molecule under analysis. For SPECenv the
problems represent .... The different problem sizes within SPEC
HPC2002 give the application users more information about machine
performance as it relates to the type of computational work they do.
-
Q15. What does the user of the SPEC HPC2002 suite have to
provide?
-
Briefly, you need a Unix or Linux system (Windows is not yet
supported) with 2 GB of memory, up to 100GB of disk, and a set of
compilers. These requirements depend on the data set that you wish to
use. Please see the details in the file system_requirements.txt of
the benchmark distribution.
-
Q16. What are the basic steps in running the
benchmarks?
-
Installation and use are covered in detail in the SPEC HPC2002 User
Documentation. The basic steps are as follows:
-
Install SPEC HPC2002 from media by running the installation
scripts to set up the appropriate directory structure and
install (and build, if necessary) the SPEC HPC2002 benchmark
run tools.
-
Determine which benchmark you wish to run.
-
Read the Run and Reporting Rules to ensure that you understand the
rules for generating that metric.
-
Create a configuration file according to the rules for that metric.
In this file, you specify compiler flags and other system-dependent
information. Benchmark- and vendor-specific example configuration
files are provided as a template for creating an initial config
file. After you become comfortable with this, you can read other
documentation to see how you can use the more complex features of
the SPEC HPC2002 tools.
-
Run the SPEC tools to build (compile), run and validate the
benchmarks.
-
If the above steps are successful, the tools will have generated a
report based on the run times and metric equations.
-
Q17: What if the tools cannot be run or built on a system? Can
they be run manually?
-
To generate SPEC-compliant results, the tools used must be approved
by SPEC. If several attempts at using the SPEC tools are not
successful for the operating system for which you purchased HPC2002,
you should contact SPEC for technical support. SPEC will work with
you to correct the problem and/or investigate SPEC-compliant
alternatives.
-
Q18. What metrics can be measured?
-
The HPC2002 suite can be used to measure and calculate the following
metrics:
-
SPECseis<size>2002
-
SPECchem<size>2002
-
SPECenv<size>2002
where <size> indicates the data size: S, M, L, X
All metrics are computed from the overall wallclock execution time T
of the benchmarks as 86400/T. This can be interpreted as the number
of times the benchmark could run consecutively in a day. Note,
however that this is not a throughput measure.
A higher score means "better performance" on the given
workload. The performance for the different data sets cannot be
compared. They may exercise different execution paths in the
benchmarks.
-
Q19. Which SPEC HPC2002 metric should be used to compare
performance?
-
It depends on your needs. SPEC provides the benchmarks and results as
tools for you to use. You need to determine how you use a computer or
what your performance requirements are and then choose the
appropriate SPEC benchmark.
-
Q20: How can I obtain SPEC HPC2002?
-
Information on ordering SPEC HPC2002 is available from the SPEC web
site (http://www.spec.org/order.html)
or by sending e-mail to info@spec.org.
-
Q21. Where are SPEC HPC2002 results available?
-
Results for all measurements submitted to SPEC are available at http://www.spec.org/hpg/
-
Q22: Can SPEC HPC2002 results be published outside of the SPEC
web site?
-
Yes, SPEC HPC2002 results can be freely published if all the run and
reporting rules have been followed and reviewed by SPEC/HPG for a
nominal fee. The SPEC HPC2002 license agreement binds every purchaser
of the suite to the run and reporting rules if results are quoted in
public. A full disclosure of the details of a performance measurement
must be provided to anyone who asks. See the SPEC HPC2002 run and
reporting rules for details.
SPEC strongly encourages that results be submitted for the web site,
since it ensures a peer review process and uniform presentation of
all results. The run and reporting rules contain an exemption clause
for research and academic use of SPEC HPC2002. Results obtained in
this context need not comply with all the requirements for other
measurements. It is required, however, that research and academic
results be clearly distinguished from results submitted officially to
SPEC.
-
Q23. Why use SPEC HPC2002?
-
SPEC HPC2002 provides the most realistic and comprehensive benchmarks
for measuring a computer system as a whole. The benchmark
applications include large, realistic, computational applications.
Among all the SPEC suites, HPC2002 has the most flexible runrules,
allowing many code optimizations. This reflects computing practices
on high-performance systems. It allows the benchmarker to achieve the
best application performance.
Other advantages to using SPEC HPC2002:
-
Benchmark programs are developed from actual end-user applications
rather than synthetic benchmarks.
-
Multiple vendors use the suite and support it.
-
SPEC HPC2002 is portable to many platforms.
-
A wide range of results are available at http://www.spec.org/hpg/
-
The benchmarks are required to be run and reported according to a
set of rules to ensure comparability and repeatability.
-
HPC2002 allows comparison of OpenMP and MPI parallelization
paradigms.
-
Q24: What organizations were involved in developing SPEC
HPC2002?
-
SPEC HPC2002 was developed by the Standard Performance Evaluation
Corp.'s High-Performance Group (SPEC/HPG), formed in January
1994. Founding partners of SPEC/HPG include SPEC members, former
members of the Perfect Benchmarks effort, and other groups working in
the benchmarking arena. SPEC/HPG's mission has remained the same:
to maintain and endorse a suite of benchmarks that represent
real-world, high-performance computing applications. Current
sustaining members include Fujitsu Ltd, Hewlett Packard, IBM, Intel,
Silicon Graphics, and Sun Microsystems. Current associates include
Argonne National Laboratory, Centre for Scientific Computing, Duke
University, Leibniz-Rechenzentrum, NCSA - University of Illinois,
North Carolina State University, National Cheng Kung University,
National Renewable Energy Lab, National University of Singapore,
Purdue University, PC Cluster Consortium, University of Miami,
University of Minnesota, University of Pisa, University of Stuttgart,
and University of Tsukuba.
-
Q25: How were the benchmarks selected?
-
They were selected based on being the most realistic, largest
computational applications that can be distributed by SPEC.
-
Q26: What are SPEC/HPG's plans for adding applications to
SPEC HPC2002?
-
SPEC/HPG is examining additional applications used in other areas of
computational analysis running on high-performance computers.
Applications under consideration include computational fluid dynamics
(CFD), molecular dynamics, climate, ocean and weather codes. The SPEC
HPC suite is updated on a regular basis. Contributions are
encouraged.
-
Q27: Will SPEC/HPG replace applications in conjunction with
changes in industrial software code?
-
Yes. Applications in the SPEC HPC2002 suite will be reviewed on a
regular basis, and when newer versions are available, they will be
incorporated into the benchmark suite. If an application falls out of
use within its industrial area, a new, more relevant application will
be adopted to replace it.
-
Q28: Will SPEC HPC2002 include more applications written in C
or C++ in the future?
-
If a suitable application representing relevant computational work in
industry is written in C or C++, it will certainly be considered. In
fact, both applications in SPEC HPC2002 V1.0 contain components
written in C.
-
Q29: How do SPEC HPC2002 benchmarks address different parallel
architectures, such as clusters, vector systems, SMPs and
NUMA?
-
SPEC HPC2002 benchmarks can be executed in serial or parallel mode.
Due to the agreed-upon software standards for parallel systems, the
parallel implementations have been based on the message-passing
programming model MPI, and on the directive-based OpenMP API. Since
high-performance computing systems use different architectures, the
SPEC HPC2002 run rules allow for some flexibility in adapting the
benchmark application to run in parallel mode. To ensure that results
are relevant to end-users, SPEC/HPG requires that systems running
SPEC HPC2002 benchmarks adhere to the following rules:
-
they must provide a suitable environment for running typical C and
Fortran programs,
-
the system vendor must offer its implementation for general use
-
the implementation must be generally available, documented, and
supported by the vendor
-
Q30: Are SPEC HPC2002 results comparable for these different
parallel architectures?
-
Yes. Most consumers of high-performance systems are interested in
running a single important application, or perhaps a small set of
critical applications, on these high-priced machines. The amount of
time it takes to solve a particular computational analysis is often
critical to a high-performance systems user's business. For these
consumers, being able to compare different machines' abilities to
complete a relevant problem of a specific size for their application
is valuable information, regardless of the architectural features of
the system itself.
-
Q31: Are SPEC HPC2002 results comparable across workload size?
Can you compare serial results to parallel results?
-
Varying the problem size, but not the system or parallelization,
demonstrates how the application performs under a greater workload.
The definition of "workload" will be application-specific
and meaningful to users doing that sort of work. With SPECseis, for
example, larger trace files require more I/O, larger FFTs, and longer
running times. A seismic analyst will be able to use the benchmark
results to understand the ability of a machine to accomplish
mission-critical tasks. Different datasets might also exercise
different functionality of the codes, which must be considered when
interpreting scalability with respect to data size. Comparing serial
to parallel results yields significant information as well: It shows
the scalability of the test system for a specific benchmark code.
-
Q32: How will SPEC/HPG address the evolution of parallel
programming models?
-
As standards emerge for parallel programming models, they will be
reflected in the SPEC HPC2002 benchmarks. In response to the growing
acceptance of SMP architectures, for example, SPEC/HPG is developing
SAS (shared address space) parallel versions of its current
benchmarks.
-
Q33: Can SPEC HPC2002 benchmarks be run on a high-end
workstation?
-
Yes, they can be run on single-processor machines. The smaller
problem sizes are likely to be the most suitable for these systems.
-
Q34: Traditionally, SPEC has not allowed any code changes in
its benchmarks. Why are code changes allowed in SPEC HPC2002 and how
did SPEC/HPG decide what should be allowed?
-
SPEC/HPG recognizes that customers who will spend many thousands to
millions of dollars on a high-performance computer are willing to
invest additional money to optimize their production codes. In
addition to delivering more return on investment, code changes are
required because there are so many different high-performance
architectures; moving an application from one architecture to another
is far more involved than porting a single CPU code from one
workstation to another.
SPEC/HPG realized that since all customers optimize their programs,
vendors should be allowed to perform the same level of optimization
as a typical customer. There are specific rules that vendors must
follow in optimizing codes. These rules were chosen to allow each
vendor to show what its systems are capable of without allowing large
application rewrites that would compromise performance comparisons.
Each vendor's code changes must be fully disclosed to the entire
SPEC/HPG membership and approved before results are published. These
changes must also be included in published reports, so customers know
what changes they would have to make to duplicate results.
-
Q35: Do SPEC HPC2002 benchmarks measure speed or
throughput?
-
Both. SPEC HPC2002 benchmarks measure the time it takes to run an
application on the system being tested -- that's a test of speed.
The SPEC HPC2002 metric also normalizes the benchmark's elapsed
time to the number of seconds in a day. So, the benchmarks also
measure throughput, since the metric reports how many benchmarks
could be run, back to back, in a given 24-hour period.
-
Q36: Does SPEC HPC2002 make SPEC CPU2000 obsolete? What does it
measure that SPEC CPU2000 does not?
-
SPEC HPC2002 results provide information that supplements SPEC
CPU2000 results. Consumers of high-performance computing systems
usually run a particular application or set of applications. It is
important for these consumers to know how applications in their area
of analysis will perform on the systems under consideration. This is
the kind of specific information that SPEC HPC2002 provides.
-
Q37: Why doesn't SPEC/HPG define a metric such as M/FLOPS
or price/performance?
-
SPEC/HPG chose to focus on total application performance for large,
industrially relevant applications. Within this benchmarking
environment, a simple metric such as M/FLOPS is inadequate and
misleading. Customers need to understand the expected performance of
systems under consideration for purchase. Real-world performance
includes all of the set-up, computation and post-processing work.
Since the pre- and post-processing phases of applications can be
significant factors in total system performance, SPEC/HPG chose to
concentrate on total system performance.
-
Q38. Why does HPC2002 only have a "peak" but no
baseline metric?
-
In contrast to other SPEC benchmark suites, SPEC HPC2002 includes
only one metric per code and data size. There are no "base"
results that would measure compiler-only performance. The SPEC
HPC2002 runrules allow certain hand optimizations for all metrics.
Since high-performance computer customers are willing to invest
programming time to tune the applications that run on their systems,
a baseline result has little meaning to them. Also, the architectures
employed in the high-performance computing market are far more
diverse than those found in single-CPU workstations. The baseline or
"out-of-the-box" performance of any given application has
no correlation to the actual performance a customer could expect to
achieve on a particular architecture.
-
Q39: Why is there no reference machine for performance
comparisons?
-
Reference machines give benchmark users a framework for judging
metrics that would otherwise just be meaningless sets of numbers.
SPEC HPC2002 uses time as its reference, not the speed of a
particular machine. The metric itself tells how many successive
benchmark runs can be completed in a 24-hour period on the system
being tested.
-
Q40: Why doesn't SPEC HPC2002 provide a composite or
aggregate performance metric?
-
Providing a composite or aggregate performance metric would undermine
the purpose of SPEC HPC2002. SPEC HPC2002 is designed to inform users
about how industrial-strength applications in their fields of
analysis will perform. These users are particularly interested in how
well their applications will scale as parallelism increases. This is
why SPEC HPC2002 reporting pages provide metrics for systems with
different numbers of processors running the same application and
problem size.
-
Q41. Some of the benchmark names may sound familiar; are these
comparable to other programs?
-
Many of the SPEC benchmarks have been derived from publicly available
application programs and all have been developed to be portable to as
many current and future hardware platforms as practical. Hardware
dependencies have been minimized to avoid unfairly favoring one
hardware platform over another. For this reason, the application
programs in this distribution should not be used to assess the
probable performance of commercially available, tuned versions of the
same application. The individual benchmarks in this suite might be
similar, but NOT identical to benchmarks or programs with the same
name which are available from sources other than SPEC. It is not
valid to compare SPEC HPC2002 benchmark results with anything other
than other SPEC HPC2002 benchmark results. (Note: This also means
that it is not valid to compare SPEC HPC2002 results to older SPEC
benchmarks; these benchmarks have been changed and should be
considered different and not comparable.)
-
Q42: What is the difference between SPEC HPC2002 and other
benchmarks for high-performance systems?
-
The most important distinction is that SPEC HPC2002 includes
applications used in industry and research to do real work. These
applications normally run on multiprocessing systems and require the
larger computing resources offered by high-end systems. Only minimal
modifications were made to the applications used in the SPEC HPC2002
suite. By leaving even "uninteresting" functions of the
application code intact, SPEC HPC2002 provides a realistic measure of
real-world application performance. SPEC/HPG's methodology
differs from previous benchmarking efforts, which concentrated only
on more numerically intensive algorithms.
A second distinction is that SPEC HPC2002 targets all
high-performance computer architectures. The applications in the
suite are currently in use on a wide variety of systems, including
workstations, clusters, SMPs, vector systems and MPPs. The
programming models used in the SPEC HPC2002 application codes --
message-passing, shared-memory parallel, and serial models -- can be
run on all of today's high-performance systems. Other benchmarks
tend to be biased towards either distributed-memory, shared-memory or
vector architectures.
Finally, SPEC HPC2002 provides more than just peak performance
numbers. To ensure that SPEC HPC2002 reflects performance for real
applications, only a limited number of optimizations are allowed.
This contrasts with benchmarks that allow a large number of
optimizations requiring unrealistic development efforts to reproduce
benchmark results. It also contrasts with benchmarks that restrict
optimizations altogether.
-
Q43: How does SPEC HPC2002 compare to the NAS parallel
benchmarks or to Parkbench?
-
The NPB (NAS Parallel Benchmarks) and Parkbench are kernels or
subsets of applications; they are used to compare architectural
implementations of machines. SPEC HPC2002 benchmarks are complete,
real-world applications used by numerous organizations to solve real
problems. These new benchmarks allow users to determine how well a
given system performs for the entire spectrum of factors needed to
solve real-world problems, including numerical computation, I/O,
memory access, software systems, and many others.
-
Q44. How do I contact SPEC for more information or technical
support?
-
SPEC can be contacted in several ways. For general information,
including other means of contacting SPEC, please see SPEC's web
site at: http://www.spec.org/
General questions can be emailed to: info@spec.org
HPC2002 Technical Support Questions can be sent to: HPC2002support@spec.org
-
Q45. Now that I've read this document, what should I do
next?
-
If you have arrived here by starting at the benchmark
distribution's readme1st document, you should now verify that
your system meets the requirements described in
system_requirements.txt and then you can install the suite,
following the instructions in install_guide_unix.txt (should
work for linux).
|
