|
Benchmarks
Tools
Order Benchmarks
SPEC
Mirror Sites
Resources
|
SPEC's Benchmarks and Tools
Current Benchmarks
Cloud
- SPEC Cloud® IaaS 2018
[benchmark info] [published
results] [order benchmark]
The SPEC Cloud® IaaS 2018 benchmark builds on the original 2016 release, updates
metrics, and workloads and adds easier setup. The benchmark stresses the
provisioning, compute, storage, and network resources of
infrastructure-as-a-service (IaaS) public and private cloud platforms
with multiple multi-instance workloads. SPEC selected the social media
NoSQL database transaction and K-Means clustering using Cassandra and
Hadoop as two significant and representative workload types within cloud
computing. For use by cloud providers, cloud consumers, hardware vendors,
virtualization software vendors, application software vendors, and
academic researchers.
CPU
- SPEC CPU® 2017
[benchmark info] [published
results] [support] [order
benchmark]
Designed to provide performance measurements that can be used to compare compute-intensive
workloads on different computer systems, the SPEC CPU® 2017 benchmark suite contains 43 benchmarks organized
into four suites: the SPECspeed® 2017 Integer suite, the SPECspeed® 2017 Floating
Point suite, the SPECrate® 2017 Integer suite, and the SPECrate® 2017 Floating
Point suite. The suite also includes an optional metric for measuring energy consumption.
Graphics and Workstation
Performance
High Performance Computing:
OpenMP, MPI, OpenACC, OpenCL
- SPECaccel® 2023
[benchmark info] [published
results] [support] [order
benchmark]
The SPECaccel® 2023 benchmark suite tests performance with computationally intensive parallel applications
running under the OpenACC and OpenMP target offloading APIs. The suite exercises the performance of the
accelerator, host CPU, memory transfer between host and accelerator, support libraries and drivers, and compilers.
- SPEC ACCEL®
[benchmark info] [published
results] [support] [order
benchmark]
The SPEC ACCEL® benchmark suite tests performance with a suite of computationally intensive
parallel applications running under the OpenCL 1.1, OpenACC 1.0, and
OpenMP 4.5 APIs. The suite exercises the performance of the accelerator,
host CPU, memory transfer between host and accelerator, support libraries
and drivers, and compilers.
- SPEChpc™ 2021
[benchmark info][published results]
[support][order benchmark]
The SPEChpc™ benchmark contains four suites, Tiny, Small, Medium, and Large, including groups of full applications
or mini-apps covering a wide range of scientific domains and Fortran/C/C++ programming languages.
Each suite uses increasingly larger workloads to allow for appropriate evaluation of HPC systems at
different sizes, ranging from a single node to hundreds of nodes. All benchmarks are ported to use
either pure-MPI or hybrid MPI+OpenACC, MPI+OpenMP (task/thread based), or MPI+OpenMP using "Target",
thus allowing measurement on heterogenous system.
- SPEC MPI® 2007
[benchmark info] [published
results] [support] [order
benchmark]
The SPEC MPI® 2007 benchmark suite evaluates MPI-parallel, floating
point, compute intensive performance across a wide range of cluster
and SMP hardware. The suite consists of the intial MPIM2007 suite
and MPIL2007, which contains larger working sets and longer run
times than MPIM2007. All benchmarks in the suite are developed in
compliance with MPI 2.1 standard.
- SPEC OMP® 2012
[benchmark info] [published
results] [support] [order
benchmark]
The SPEC OMP® 2012 benchmark suite is the successor to
the SPEC OMP® 2001 benchmark suite, designed for measuring performance
using applications based on the OpenMP 3.1 standard for shared-memory
parallel processing. The suite also includes an optional metric for
measuring energy consumption.
Java Client/Server
- SPECjbb® 2015
[benchmark info] [published
results] [support]
[order benchmark]
The SPECjbb® 2015 benchmark has been developed from the ground up to measure
performance based on the latest Java application features. It is relevant to all audiences
who are interested in Java server performance, including JVM vendors, hardware developers,
Java application developers, researchers and members of the academic community.
- SPECjEnterprise® 2018 Web Profile
[benchmark info] [published
results] [support]
[order benchmark]
The SPECjEnterprise® 2018 Web Profile benchmark measures full-system performance for Java
Enterprise Edition (Java EE) Web Profile Version 7 or later application
servers, databases and supporting infrastructure.
- SPECjEnterprise® 2010
[benchmark info] [published
results] [support]
[order benchmark]
The SPECjEnterprise® 2010 benchmark measures full-system performance for Java Enterprise
Edition (Java EE) 5 or later application servers, databases and
supporting infrastructure and expands the scope of the SPECjAppServer® 2004 benchmark.
- SPECjvm® 2008
[benchmark info] [published
results] [support]
[download benchmark]
The SPECjvm® 2008 benchmark suite is designed for measuring the performance of
a Java Runtime Environment (JRE), containing several real life applications
and benchmarks focusing on core java functionality. The benchmark
workload mimics a variety of common general purpose application
computations.
Storage
- SPECstorage® Solution 2020
[benchmark info] [published
results] [support]
[order benchmark]
The SPECstorage® Solution 2020 benchmark is the most recent version of SPEC's
benchmark suite designed to evaluate performance using file server throughput and response time.
Power
- SPECpower_ssj® 2008
[benchmark info] [published
results] [support]
[order benchmark]
The SPECpower_ssj® 2008 benchmark is the first industry-standard SPEC benchmark
that evaluates the power and performance characteristics of volume
server class computers. The initial benchmark addresses the performance
of server-side Java, and additional workloads are planned.
- Other SPEC benchmarks incorporating power measurement
Virtualization
- SPECvirt® Datacenter 2021
[benchmark info] [published
results] [support] [order
benchmark]
The SPECvirt® Datacenter 2021 benchmark is the third generation SPEC
benchmark for evaluating the performance of virtualized environments
and the first for measuring the performance of multiple hosts in
virtualized datacenters.
- SPEC VIRT_SC® 2013
[benchmark info] [published
results] [support] [order
benchmark]
SPEC's updated benchmark addressing performance evaluation of datacenter
servers used in virtualized server consolidation. The SPEC VIRT_SC® 2013
benchmark measures the end-to-end performance of all system components including
the hardware, virtualization platform, and the virtualized guest
operating system and application software. The benchmark supports
hardware virtualization, operating system virtualization, and hardware
partitioning schemes.
SPEC Tools
- SPEC SERT® Suite 2.0
[benchmark info] [support]
[order software]
The SERT® suite was created by Standard
Performance Evaluation Corporation (SPEC) at the request of the
US Environmental Protection Agency. The SERT suite 2.0 adds a single-value
metric, reduces runtime, improves automation and testing, and broadens
device and platform support. Designed to be simple to configure and use
via a comprehensive graphical user interface, the SERT suite uses a set
of synthetic worklets to test discrete system components such as processors,
memory and storage, providing detailed power consumption data at different
load levels.
The SERT suite metric, created with the support of the RG Power Working Group,
rates the server efficiency of single- and multi-node servers across a broad
span of configurations.
- SPEC SERT® Suite 1.1.1
[benchmark info] [support]
[order software]
The SERT® suite was created by Standard Performance
Evaluation Corporation (SPEC) at the request of the US Environmental Protection
Agency. The SERT suite 1.1.1 is the most current SERT version supported by the U.S. EPA
Energy Star v2.0 program. Designed to be simple to configure and use via a comprehensive
graphical user interface, the SERT suite uses a set of synthetic worklets to test
discrete system components such as processors, memory and storage, providing detailed
power consumption data at different load levels.
- SPEC Chauffeur® WDK Tool
[kit info] [order software]
The Chauffeur® WDK (Worklet Development Kit) Tool was designed to simplify the development of workloads
for measuring both performance and energy efficiency. Because the Chauffeur
WDK tool contains functions that are common to most workloads, developers
of new workloads can focus on the actual business logic of the application,
and take advantage of the Chauffeur WDK tool's capabilities for configuration,
run-time, data collection, validation, and reporting.
The Chauffeur
WDK tool was initially designed to meet the requirements of the
SERT. However, SPEC recognized that the framework would also be
useful for research and development purposes. The Chauffeur framework
is now being made available as the Chauffeur WDK (Worklet Development
Kit). This kit can be used to develop new workloads (or "worklets"
in Chauffeur terminology). Researchers can also use the Chauffeur WDK to configure
worklets to run in different ways, in order to mimic the behavior
of different types of applications. These features can be used in
the development and assessment of new technologies such as power
management capabilities.
Version 2.0 is based on the SERT suite 2.0 infrastructure
and includes significant enhancements to the hardware detection,
customization options of generating HTML reports, and developer
documentation. It has now reduced memory requirements for the Director
when signing results files and reduced the size of the result output
for large systems or clusters.
The Chauffeur WDK tool 2.0 added worklet-specific normalization of results
and an updated list of supported operating systems including Ubuntu
(14.04 LTS and 16.04 LTS) as well as current versions of Windows Server,
RHEL, SLES, AIX, and Solaris. The Chauffeur WDK tool also includes the latest PTDaemon
integration for power analyzers and temperature sensors, along with data
collection, validation and reporting.
- PTDaemon®
[info]
The power temperature daemon (also known as PTDaemon®) is used to offload
the work of controlling a power analyzer or temperature sensor during
measurement intervals to a system other than the SUT. It hides the details
of different power analyzer interface protocols and behaviors from the
benchmark software, presenting a common TCP/IP-based interface that can
be readily integrated into different benchmark harnesses. Benchmarks already
using PTDaemon include SPECpower_ssj 2008, SPEC CPU 2017, SPEC VIRT_SC 2013,
the SERT suite, and the Chauffeur WDK.
Retired Benchmarks
|
