SPEC Cloud IaaS 2016 Benchmark
Copyright © 2016 Standard Performance Evaluation Corporation
Cloud Vendor: Dell Inc.
Cloud Type / SUT Type: private/whitebox
Hardware Platform: x86_64
Hypervisor: KVM
Cloud Infrastructure: Red Hat Enterprise Linux Openstack Platform 7
Scalability: 10.3 @ 10 Application Instances
Elasticity: 63.0%
Mean Instance Provisioning Time: 163s
Tested by: Dell Opensource Solutions SPEC Licence Number : 999 Test Date : Apr-2016
Performance Sections
Performance Summary
Performance Details
Validation, Errors, and Issues
Glossary of Terms
SUT Configuration Sections
Instance Configuration
Cloud Configuration
Network Configuration
Storage Configuration
Elasticity + Scalability Phase Date/Time and Test Region
Elasticity + Scalability Start Time: 2016-03-05_17:33:51_UTC
Elasticity + Scalability End Time: 2016-03-05_18:19:59_UTC
Test Region: US Central Time Zone
Cloud Informational Metrics
AI Provisioning Success: 100.00%
AI Run Success: 100.00
Total Instances: 65
Reference ID: AI17-rc4p2-1

Performance Summary

Baseline Summary Results for YCSB
Throughput (ops/s) Insert Latency 99% (ms) Read Latency 99% (ms) AI Provisioning Time (s)
Average (across 5) 13082.6 2.048 4.597 147.4

Baseline Summary Results for KMeans
Completion Time (s) AI Provisioning Time (s)
Average (across 5) 115.7 122.0

Scalability/Elasticity Summary Results for YCSB for 5 Valid AIs
Av. Throughput (ops/s) Av. Insert Latency 99% (ms) Av. Read Latency 99% (ms) Av. Provisioning Time (s) Scalability Elasticity (%)
9480.9 3.402 5.831 220.6 5.80 69.95

Scalability/Elasticity Summary Results for KMeans for 5 Valid AIs
Av. Completion Time (s) Av. Provisioning Time (s) Scalability Elasticity (%)
211.5 204.2 4.54 55.97

YCSB Summary Results by AI
AI name Run count Run code Av. Throughput (ops/s) Av. Insert Latency 99% (ms) Av. Read Latency 99% (ms) AI. Provisioning time (s) AI. Prov. Initiated from Elasticity Start (s)
ai_11 7 0 10758.6 2.641 4.591 203.0 1.0
ai_14 5 0 10408.9 2.939 5.315 186.0 535.0
ai_15 4 0 9132.2 2.594 4.503 220.0 838.0
ai_17 2 0 9892.1 2.570 5.710 217.0 1433.0
ai_19 1 0 7212.4 6.267 9.039 277.0 1988.0

KMeans Summary Results by AI
AI name Run count Run code Av. Completion Time (s) Av. Provisioning time (s) AI. Prov. Initiated from Elasticity Start (s)
ai_12 9 0 194.4 162.0 7.0
ai_13 5 0 250.4 213.0 528.0
ai_16 4 0 221.0 162.0 966.0
ai_18 2 0 218.5 209.0 1491.0
ai_20 1 0 173.0 275.0 2054.0


Validation, Errors, and Issues

Glossary of "Run code": 0 no error, 3 if an AI failed to provision, 2 if an AI that was provisioned but did not report any runs when the benchmark terminated - the AIs that are provisioning towards the end of elasticity + scalability phase may have zero runs, 1 for all other errors including AIs that were provisioned but did not report any result or reported results after QoS limits.

AIs that were issued but their provisioning was inprogress when the benchmark terminated are not shown.


Instance Configuration

Instance Configuration Hadoop Master
Parameter Value
instance_type_name m1.cmedium3
cpu 2
memory 12GB
disk_size 40GB
disk_backed block storage
image_name cb_speccloud_kmeans_hadoop_271_centos-raw
Notes Hadoop 2.7.1 / Centos 7.1

Instance Configuration Hadoop Slave
Parameter Value
instance_type_name m1.clarge3
cpu 4
memory 20GB
disk_size 80GB
disk_backed block storage
image_name cb_speccloud_kmeans_hadoop_271_centos-raw
Notes Hadoop 2.7.1 / Centos 7.1

Instance Configuration YCSB
Parameter Value
instance_type_name m1.cmedium3
cpu 2
memory 12GB
disk_size 40GB
disk_backed block storage
image_name cb_speccloud_cassandra_2111_centos-raw
Notes Cassandra 2.11.1 / Centos 7.1

Instance Configuration Cassandra
Parameter Value
instance_type_name m1.cmedium3
cpu 2
memory 12GB
disk_size 40GB
disk_backed block storage
image_name cb_speccloud_cassandra_2111_centos-raw
Notes Cassandra 2.11.1 / Centos 7.1


Cloud Configuration

Cloud High Level Information
Parameter Value
Name Openstack
Type private
Sut_Type whitebox
Virtualized KVM
Manager_Protocol Red Hat Enterprise Linux
Manager_Version Openstack Platform 7
Hardware_Availability May-2012
Software/Service_Availability Aug-2015
Geographic_Distribution 1
Notes Cloud infrastructure and management nodes all reside in a single datacenter located in Austin, TX.

Cloud Under Test Software Levels
Parameter Value
Python_version 2.7.6
JVM_version 1.7.0_45
Hadoop_version 2.7.1
YCSB_version 0.4.0
HiBench_version 2.2
Cassandra_version 2.1.11
Notes N/A

Benchmark Harness Node
Parameter Value
Vendor Dell
Platform PowerEdge R720xd
CPU_Description Intel Xeon, E5-2640 0
CPU_Frequency_MHz 2500
RAM_Memory 48 GB, ECC, DDR3, 1600MHz RDIMM
Local_Storage 8x 1TB, SATA, RAID50, 7200 RPM
Quantity 1
Availability_Date May-2012
OS_Version RHEL 7.1
OS_Availability_Date March-2015
Network_Adapter_1 1 x 1 x Broadcom Corporation NetXtreme BCM5720 4P
Network_Adapter_2 2 x Intel X520 10Gb 2P SFP+
Network_port_type_1 1000BASE-T
Network_port_type_2 10GBASE-SR
Network_speed_port_1 1 Gb/s
Network_speed_port_2 10 Gb/s
Notes Solution Admin Host (SAH) hosts the benchmark driver harness

Cloud Manager Node
Parameter Value
Hw_Vendor_node1,3 Dell
Hw_Platform_node1,3 PowerEdge R720xd
CPU_Type_node1,3 Intel Xeon, E5-2640 0
CPU_Count_node1,3 4
CPU_Cores_Per_node1,3 6
CPU_Frequency_MHz_node1,3 2500
RAM_Memory_GB_node1,3 96,48
Primary_Cache_KB_node1,3 190 KB I + 1500 KB D per core, on chip
Secondary_Cache_KB_node1,3 1500 KB I+D per core, on chip
RAM_Memory_Type_node1,3 ECC DDR3 1333 MHz
Hw_Availability_Date_node1,3 May-2012
Local_Storage_node1,3 24x1 TB HDD, RAID 50, 7200 RPM
Quantity_node1,3 2
Virtualized_node1,3 baremetal
Network_Adapter_node1,3_1 1 x Broadcom Corporation NetXtreme BCM5720 4P
Network_Adapter_node1,3_2 2 x Intel X520 10Gb 2P SFP+
Network_port_type_node1,3_1 1000BASE-T
Network_port_type_node1,3_2 10GBASE-SR
Network_speed_port_node1,3_1 1 Gb/s
Network_speed_port_node1,3_2 10 Gb/s
OS_Version_node1,3 RHEL 7.1
OS_Availability_Date_node1,3 March-2015
Hw_Vendor_node2 Dell
Hw_Platform_node2 PowerEdge R720xd
CPU_Type_node2 Intel Xeon, E5-2670 v2
CPU_Count_node2 2
CPU_Cores_Per_node2 10
CPU_Frequency_MHz_node2 2500
RAM_Memory_GB_node2 96
Primary_Cache_KB_node2 310 KB I + 2500 KB D per core, on chip
Secondary_Cache_KB_node2 2500 KB I+D per core, on chip
RAM_Memory_Type_node2 ECC DDR3 1600 MHz
Hw_Availability_Date_node2 May-2012
Local_Storage_node2 24x1 TB HDD, RAID 50, 7200 RPM
Quantity_node2 1
Virtualized_node2 baremetal
Network_Adapter_node2_1 1 x Broadcom Corporation NetXtreme BCM5720 4P
Network_Adapter_node2_2 2 x Intel X520 10Gb 2P SFP+
Network_port_type_node2_1 1000BASE-T
Network_port_type_node2_2 10GBASE-SR
Network_speed_port_node2_1 1 Gb/s
Network_speed_port_node2_2 10 Gb/s
OS_Version_node2 RHEL 7.1
OS_Availability_Date_node2 March-2015
Notes 3x Cloud Manager (Openstack controller) nodes

Cloud Computes Nodes
Parameter Value
type 1_Compute_nodes 5
type 1_Hw_Vendor Dell., Inc.
type 1_Hw_Platform PowerEdge R720xd
type 1_Quantity 5
type 1_CPU_Type Intel Xeon, E5-2640 0
type 1_CPU_Count 10
type 1_CPU_Cores_Per 6
type 1_CPU_Frequency_MHz 2500
type 1_RAM_Memory_GB 192
type 1_Primary_Cache_KB 190 KB I + 1500 KB D per core, on chip
type 1_Secondary_Cache_KB 15000 KB I+D per core, on chip
type 1_RAM_Memory_Type ECC DDR3 1333 MHz
type 1_Hw_Availability_Date May-2012
type 1_Local_Storage 24x 1 TB HDD, SATA, RAID 50, 7200 RPM
type 1_Virtualized baremetal
type 1_Network_Adapter_1 1 x Broadcom Corporation NetXtreme BCM5720 4P
type 1_Network_Adapter_2 2 x Intel X520 10Gb 2P SFP+
type 1_Network_port_type_1 1000BASE-T
type 1_Network_port_type_2 10GBASE-SR
type 1_Network_speed_port_1 1 Gb/s
type 1_Network_speed_port_2 10 Gb/s
type 1_OS_Version RHEL 7.1
type 1_OS_Availability_Date March-2015
type 1_Notes N/A
type 2_Compute_nodes 2
type 2_Hw_Vendor Dell., Inc.
type 2_Hw_Platform PowerEdge R720xd
type 2_CPU_Type Intel Xeon, E5-2670 v2
type 2_CPU_Count 4
type 2_CPU_Cores_Per 8
type 2_CPU_Frequency_MHz 2500
type 2_RAM_Memory_GB 192
type 2_Primary_Cache_KB 310 KB I + 2500 KB D per core, on chip
type 2_Secondary_Cache_KB 25000 KB I+D per core, on chip
type 2_RAM_Memory_Type ECC DDR3 1600 MHz
type 2_Hw_Availability_Date May-2012
type 2_Local_Storage 12x1 TB, RAID 0, 7200 RPM
type 2_Quantity 2
type 2_Virtualized baremetal
type 2_Network_port_1 1 Gbps
type 2_Network_port_2 10 Gbps
type 2_OS_Version RHEL 7.1
type 2_OS_Availability_Date Mar-2015

Cloud Network Information

Cloud Network Information
Parameter Value
network 1_Technology ethernet
network 1_Protocol TCP/IP
network 1_Protocol_version IPV4
network 1_Speed_Mbps 1000
network 1_Function management
network 1_Notes network_1 related notes
network 2_Technology ethernet
network 2_Protocol TCP/IP
network 2_Protocol_version IPV4
network 2_Speed_Mbps 10000
network 2_Function data
network 2_Notes network_2 related notes

Cloud Schematic


Storage Configuration

Cloud Storage
Parameter Value
Storage 1_Attach_Type network
Storage 1_Capacity 72 TB
Storage 1_Technology HDD
Storage 1_Notes Ceph Storage Nodes 3 x R720xd



Performance Details

YCSB Detailed Results for Application Instance Runs
AI name Run id Throughput (ops/s) Insert Latency 99% (ms) Read Latency 99% (ms) Data gen time (s) AI Run Errors NTP Error
ai_11 1 13389.6 2.085 4.987 112.0 0 0
2 9972.6 2.205 3.509 119.0 0 0
3 10945.2 2.097 4.073 112.0 0 0
4 11309.5 2.881 5.771 166.0 0 0
5 9444.5 2.487 3.379 132.0 0 0
6 10049.2 3.787 5.819 255.0 0 0
7 10199.9 2.947 4.599 323.0 0 0
AI name Run id Throughput (ops/s) Insert Latency 99% (ms) Read Latency 99% (ms) Data gen time (s) AI Run Errors NTP Error
ai_14 1 9886.9 2.641 6.443 169.0 0 0
2 11619.0 2.133 3.055 147.0 0 0
3 10077.3 2.533 4.639 255.0 0 0
4 9285.6 4.683 8.407 175.0 0 0
5 11175.7 2.703 4.029 226.0 0 0
AI name Run id Throughput (ops/s) Insert Latency 99% (ms) Read Latency 99% (ms) Data gen time (s) AI Run Errors NTP Error
ai_15 1 12779.9 2.197 6.479 110.0 0 0
2 8734.2 3.007 3.885 210.0 0 0
3 6219.6 2.195 3.283 209.0 0 0
4 8795.2 2.977 4.363 322.0 0 0
AI name Run id Throughput (ops/s) Insert Latency 99% (ms) Read Latency 99% (ms) Data gen time (s) AI Run Errors NTP Error
ai_17 1 9981.8 2.763 8.091 234.0 0 0
2 9802.4 2.377 3.329 328.0 0 0
AI name Run id Throughput (ops/s) Insert Latency 99% (ms) Read Latency 99% (ms) Data gen time (s) AI Run Errors NTP Error
ai_19 1 7212.4 6.267 9.039 271.0 0 0

KMeans Detailed Results for Application Instance Runs
AI name Run id Completion Times (s) Data gen time (s) AI Run Errors NTP Error
ai_12 1 105.0 35.0 0 0
2 154.0 59.0 0 0
3 92.0 126.0 0 0
4 164.0 37.0 0 0
5 200.0 71.0 0 0
6 143.0 56.0 0 0
7 189.0 108.0 0 0
8 254.0 88.0 0 0
9 449.0 149.0 0 0
AI name Run id Completion Times (s) Data gen time (s) AI Run Errors NTP Error
ai_13 1 313.0 69.0 0 0
2 176.0 76.0 0 0
3 183.0 200.0 0 0
4 323.0 85.0 0 0
5 257.0 132.0 0 0
AI name Run id Completion Times (s) Data gen time (s) AI Run Errors NTP Error
ai_16 1 182.0 64.0 0 0
2 343.0 50.0 0 0
3 100.0 101.0 0 0
4 259.0 77.0 0 0
AI name Run id Completion Times (s) Data gen time (s) AI Run Errors NTP Error
ai_18 1 194.0 113.0 0 0
2 243.0 146.0 0 0
AI name Run id Completion Times (s) Data gen time (s) AI Run Errors NTP Error
ai_20 1 173.0 85.0 0 0


Glossary of terms (sorted alphabetically)

Application Instance

A group of instances created to run a single workload together. An application instance comprises a workload driver instance and set of instances, which are stressed by the workload driver. SPEC Cloud IaaS 2016 benchmark uses multiple application instances during specific phases to determine elasticity and scaling.

Baseline phase

In baseline, peak performance for each workload is determined in five separate test runs. During each workload run, instances are provisioned, data set is generated, load generator is started, results are accumulated, and the instances are destroyed. The workloads are run in a sequential fashion. Data from the baseline phase is used to establish parameters for the Elasticity + Scalability phase.

Benchmark phases

Benchmark has two phases, namely baseline and elasticity + scalability.

Blackbox Cloud

A cloud-provider provides a general specification of the SUT, usually in terms of how the cloud consumer may be billed. The exact hardware details corresponding to these compute units may not be known. This will typically be the case if the entity benchmarking the cloud is different from a cloud provider.

Cloud (system or service)

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

Cloud Consumer

A person or organization that is a customer of a cloud; note that a cloud customer may itself be a cloud and that clouds may offer services to one another.

Cloud Provider

An organization that provides cloud services to customers who pay for only the computing time and services actually used.

Elasticity

Elasticity measures whether the work performed by application instances scales linearly in a cloud. That is, for statistically similar work, the performance of N application instances in a cloud must be the same as the performance of application instances during baseline phase when no other load is introduced by the tester. Elasticity is expressed as a percentage (out of 100).

Elasticity + scalability phase

In the Elasticity + Scalability phase, new application instances are created, and they run the workloads concurrently to determine the elasticity and scalability metrics. The benchmark reports are generated at the end of elasticity + scalability phase.

Infrastructure as a Service (IaaS)

The Service Provider gives the Cloud-Consumer the capability to the provision processing, storage, network, and basic computing resources. They can also deploy and run arbitrary operating systems. The End-Consumer does not manage or control the underlying physical cloud infrastructure, but has control over the operating system, assigned storage, deployed applications, and limited control of select networking components (e.g., host firewalls).

Instance

An instance is an abstracted execution environment which presents an operating system (either discrete or virtualized). The abstracted execution environment presents the appearance of a dedicated computer with CPU, memory and I/O resources available to the operating system. In SPEC Cloud, an instance consists of a single OS and the application software stack that supports a single SPEC Cloud component workload. There are several methods of implementing an instance, including physical machines, virtual machines, or containers. An instance is created or destroyed using an API provided by an IaaS cloud.

Instance image

An image on the disk from which an instance is provisioned. Common formats for instance image include QCOW2 (Qemu copy on write 2), RAW, or AMI (Amazon machine image)

Physical machine

A set of connected components consisting on one or more general purpose processors (CPUs), memory, network connectivity, and mass storage either local (disk) or remote (network attached, block storage). The physical machine can have its own physical packaging or be installed as a blade in a blade chassis. An example would be a multi-core server with 4 GB of memory, 250 GB disk, and 1 Gb/s network adapter.

Provisioning Time

The measured time needed to bring up a new instance, or add more resources (like CPU or storage) to an existing instance. [Reference: CloudWhitePaper]

Within this document, this will be divided into two measurements:

1) Instance: The time from request to create a new instance until that instance responds to a netcat probe on port 22.

2) Application instance: The time from request to create a new instance until the associated cluster reports readiness to accept client requests. YCSB (Yahoo! Cloud Serving Benchmark) is ready when all nodes in the underlying database are part of the Cassandra cluster. K-Means is ready when the all nodes in HDFS are ready and part of the HDFS cluster.

Private Cloud

The cloud infrastructure is provisioned for exclusive use by a single organization comprising single or multiple consumers (e.g., business units). It may be owned, managed, and operated by the organization, a third party, or some combination of them, and it may exist on or off premises.

Public Cloud

The cloud infrastructure is provisioned for open use by the general public. It may be owned, managed, and operated by a business, academic, or government organization, or some combination of them. It exists on the premises of the cloud provider.

Quality of Service (QoS)

The minimum percent (e.g., 95%) of collected values that complete within a predefined threshold in each measurement category. The Run and Reporting Rules document contains specific QoS limitations for each workload and benchmark.

Response Time

The time between when the work item request is issued until the corresponding completion condition. This definition is identical to the YCSB Latency metric.

Scalability

Scalability measures the total amount of work performed by application instances running in a cloud. The aggregate work performed by one or more application instances should linearly scale in an ideal cloud. It answers the question: How much more gets done if N instead of one (1) application instances? Ideally, each additional application instance will contribute 100% of its individual capability, but this may not be the case as the cloud gets loaded with multiple application instances or other workloads (for public clouds). Scalability is reported as a unit-less number @ the number of valid application instances. The number is an aggregate of workloads metrics across all application instances running in a cloud normalized by workload metrics in a reference cloud platform. The reference platform metrics are an average of workload metrics taken across multiple cloud platforms during the benchmark development.

SUT

One or more cloud services under test. This includes all hardware, network, base software and management systems used for the cloud service.

Variability

The difference between different runs of the benchmark. Typically, variability arises due to factors such as multi-tenancy and time of day execution, and may be more pronounced on public clouds.

Whitebox Cloud

The SUT's exact engineering specifications including all hardware and software are known and under the control of the tester. This will typically be the case for private clouds.