Performance Summary

Validation, Errors, and Issues

• Stopping condition reason: Benchmark stopped because number of AIs reporting results reached the configured value. Result received from AIs:20, _rules_reported_ais:20, _ais_issued:23:NOK
• 0 YCSB AIs removed due to 60/40 rule
• 0 KMeans AIs removed due to 60/40 rule
• 1 YCSB AIs with zero runs
• 2 KMeans AIs with zero runs
• 0 YCSB AIs that failed to provision
• 0 KMeans AIs that failed to provision

Glossary of "Run code": 0 no error, 3 if an AI failed to provision, 2 if an AI that was provisioned but did not report any runs when the benchmark terminated - the AIs that are provisioning towards the end of elasticity + scalability phase may have zero runs, 1 for all other errors including AIs that were provisioned but did not report any result or reported results after QoS limits.

AIs that were issued but their provisioning was inprogress when the benchmark terminated are not shown.

Instance Configuration Hadoop Master
Parameter	Value
instance_type_name	m1.cmedium3
cpu	2
memory	12GB
disk_size	40GB
disk_backed	block storage
image_name	cb_speccloud_kmeans_hadoop_271_centos
Notes	Hadoop 2.7.1 / Centos 7.1

Instance Configuration Hadoop Slave
Parameter	Value
instance_type_name	m1.clarge3
cpu	4
memory	20GB
disk_size	80GB
disk_backed	block storage
image_name	cb_speccloud_kmeans_hadoop_271_centos
Notes	Hadoop 2.7.1 / Centos 7.1

Instance Configuration YCSB
Parameter	Value
instance_type_name	m1.cmedium3
cpu	2
memory	12GB
disk_size	40GB
disk_backed	block storage
image_name	cb_speccloud_cassandra_2111_centos-raw
Notes	Cassandra 2.11.1 / Centos 7.1

Instance Configuration Cassandra
Parameter	Value
instance_type_name	m1.cmedium3
cpu	2
memory	12GB
disk_size	40GB
disk_backed	block storage
image_name	cb_speccloud_cassandra_2111_centos-raw
Notes	Cassandra 2.11.1 / Centos 7.1

Cloud High Level Information and Availability
Parameter	Value
Name	Openstack
Type	private
Sut_Type	whitebox
Virtualized	KVM
Manager_Protocol	Red Hat Enterprise Linux
Manager_Version	Openstack Platform 8
Geographic_Distribution	1
Hardware_Availability	May-2016
Software/Service_Availability	April-2016
Notes	Cloud infrastructure and management nodes all reside in a single datacenter located in Austin, TX. Cloud installation followed standard Red Hat OSP Director practices as per installation configuration files found in supporting files (~/cloud_config/whitebox/cloud_mgmt_software/templates.zip) other than the configuration settings shown in the supporting files ((~/cloud_config/whitebox/cloud_mgmt_software/Openstack Configuration) there was no specialized tuning of the Openstack cloud.

Cloud Under Test Software Levels
Parameter	Value
Python_version	2.7.6
JVM_version	1.7.0_85
YCSB_version	0.4.0
Hadoop_version	2.7.1
Cassandra_version	2.1.11
Notes	N/A

Benchmark Harness Node
Parameter	Value
Vendor	Dell
Platform	PowerEdge R630
CPU_Description	Intel(R) Xeon(R) CPU E5-2650 v4
CPU_Frequency_MHz	2.2 Ghz
CPU_Count	2
CPU_Cores_Per	12
RAM_Memory_GB	128
Primary_Cache_KB	768KiB, L1 cache
Secondary_Cache_KB	3072KiB, L2 cache
Third_Cache_KB	30.00MB, L3 cache
RAM_Memory	(8 x 16GB ) 128 GB, ECC, DDR-4, 2400MHz
Local_Storage	8x 1TB, SATA, RAID50, 7200 RPM
Network_Adapter_1	2 x 82599ES 10-Gigabit SFI/SFP+
Network_Adapter_2	2 x Intel X520 10Gb 2P SFP+
Network_Adapter_3	2 x Intel I350 Gigabit Network
Network_port_1	10 Gb/s
Network_port_2	10 Gb/s
Network_port_3	1 Gb/s
Availability_Date	May-2016
OS_Version	RHEL 7.2
OS_Release_Date	Nov-2015
Notes	Solution Admin Host (SAH) hosts the benchmark driver harness

Cloud Manager Node
Parameter	Value
Hw_Vendor	Dell
Hw_Platform	PowerEdge R630
CPU_Type	Intel(R) Xeon(R) CPU E5-2650 v4
CPU_Count	2
CPU_Cores_Per	12
CPU_Frequency_MHz	2.2 Ghz
RAM_Memory_GB	128
Primary_Cache_KB	768KiB, L1 cache
Secondary_Cache_KB	3072KiB, L2 cache
Third_Cache_KB	30.00MB
RAM_Memory_Type	128GB (8x16GB), ECC DDR4 2400MHz
Hw_Release_Date	May-2016
Local_Storage	4x500GB,7.2K,2.5"
Quantity	3
Virtualized	KVM
Network_Adapter	2 x Intel I350 Gigabit
Network_port_1	10 Gb/s
Network_port_2	10 Gb/s
Network_port_3	1 Gb/s
OS_Version	RHEL 7.2
OS_Release_Date	Nov-2015
OS_Version_node1	RHEL 7.2
OS_Release_Date_node1	Nov-2015
Notes	3x Cloud Manager (Openstack controller) nodes

Cloud Computes Nodes
Parameter	Value
type 1_Compute_nodes	9
type 1_Hw_Vendor	Dell., Inc.
type 1_Hw_Platform	PowerEdge R630
type 1_CPU_Type	Intel(R) Xeon(R) CPU E5-2680 v4
type 1_CPU_Count	2
type 1_CPU_Cores_Per	14
type 1_CPU_Frequency_MHz	2.4 Ghz
type 1_RAM_Memory_GB	256
type 1_Primary_Cache_KB	901.12KiB, L1 cache
type 1_Secondary_Cache_KB	3584KiB, L2 cache
type 1_Third_Cache_KB	35.00MB, L3 cache
type 1_RAM_Memory_Type	256GB (8x32GB), ECC DDR4 2400MHz
type 1_Hw_Release_Date	May-2016
type 1_Local_Storage	6x600GB,7.2K,2.5"
type 1_Virtualized	KVM
type 1_Network_Adapter	2 x Intel I350 Gigabit
type 1_Network_port_1	10 Gb/s
type 1_Network_port_2	10 Gb/s
type 1_Network_port_3	1 Gb/s
type 1_OS_Version	RHEL 7.2
type 1_OS_Release_Date	Nov-2015
type 1_Notes	N/A

Cloud Schematic

Application Instance

A group of instances created to run a single workload together. An application instance comprises a workload driver instance and set of instances, which are stressed by the workload driver. SPEC Cloud IaaS 2016 benchmark uses multiple application instances during specific phases to determine elasticity and scaling.

Baseline phase

In baseline, peak performance for each workload is determined in five separate test runs. During each workload run, instances are provisioned, data set is generated, load generator is started, results are accumulated, and the instances are destroyed. The workloads are run in a sequential fashion. Data from the baseline phase is used to establish parameters for the Elasticity + Scalability phase.

Benchmark phases

Benchmark has two phases, namely baseline and elasticity + scalability.

Blackbox Cloud

A cloud-provider provides a general specification of the SUT, usually in terms of how the cloud consumer may be billed. The exact hardware details corresponding to these compute units may not be known. This will typically be the case if the entity benchmarking the cloud is different from a cloud provider.

Cloud (system or service)

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

Cloud Consumer

A person or organization that is a customer of a cloud; note that a cloud customer may itself be a cloud and that clouds may offer services to one another.

Cloud Provider

An organization that provides cloud services to customers who pay for only the computing time and services actually used.

Elasticity

Elasticity measures whether the work performed by application instances scales linearly in a cloud. That is, for statistically similar work, the performance of N application instances in a cloud must be the same as the performance of application instances during baseline phase when no other load is introduced by the tester. Elasticity is expressed as a percentage (out of 100).

Elasticity + scalability phase

In the Elasticity + Scalability phase, new application instances are created, and they run the workloads concurrently to determine the elasticity and scalability metrics. The benchmark reports are generated at the end of elasticity + scalability phase.

Infrastructure as a Service (IaaS)

The Service Provider gives the Cloud-Consumer the capability to the provision processing, storage, network, and basic computing resources. They can also deploy and run arbitrary operating systems. The End-Consumer does not manage or control the underlying physical cloud infrastructure, but has control over the operating system, assigned storage, deployed applications, and limited control of select networking components (e.g., host firewalls).

Instance

An instance is an abstracted execution environment which presents an operating system (either discrete or virtualized). The abstracted execution environment presents the appearance of a dedicated computer with CPU, memory and I/O resources available to the operating system. In SPEC Cloud, an instance consists of a single OS and the application software stack that supports a single SPEC Cloud component workload. There are several methods of implementing an instance, including physical machines, virtual machines, or containers. An instance is created or destroyed using an API provided by an IaaS cloud.

Instance image

An image on the disk from which an instance is provisioned. Common formats for instance image include QCOW2 (Qemu copy on write 2), RAW, or AMI (Amazon machine image)

Physical machine

A set of connected components consisting on one or more general purpose processors (CPUs), memory, network connectivity, and mass storage either local (disk) or remote (network attached, block storage). The physical machine can have its own physical packaging or be installed as a blade in a blade chassis. An example would be a multi-core server with 4 GB of memory, 250 GB disk, and 1 Gb/s network adapter.

Provisioning Time

The measured time needed to bring up a new instance, or add more resources (like CPU or storage) to an existing instance. [Reference: CloudWhitePaper]

Within this document, this will be divided into two measurements:

1) Instance: The time from request to create a new instance until that instance responds to a netcat probe on port 22.

2) Application instance: The time from request to create a new instance until the associated cluster reports readiness to accept client requests. YCSB (Yahoo! Cloud Serving Benchmark) is ready when all nodes in the underlying database are part of the Cassandra cluster. K-Means is ready when the all nodes in HDFS are ready and part of the HDFS cluster.

Private Cloud

The cloud infrastructure is provisioned for exclusive use by a single organization comprising single or multiple consumers (e.g., business units). It may be owned, managed, and operated by the organization, a third party, or some combination of them, and it may exist on or off premises.

Public Cloud

The cloud infrastructure is provisioned for open use by the general public. It may be owned, managed, and operated by a business, academic, or government organization, or some combination of them. It exists on the premises of the cloud provider.

Quality of Service (QoS)

The minimum percent (e.g., 95%) of collected values that complete within a predefined threshold in each measurement category. The Run and Reporting Rules document contains specific QoS limitations for each workload and benchmark.

Response Time

The time between when the work item request is issued until the corresponding completion condition. This definition is identical to the YCSB Latency metric.

Scalability

Scalability measures the total amount of work performed by application instances running in a cloud. The aggregate work performed by one or more application instances should linearly scale in an ideal cloud. It answers the question: How much more gets done if N instead of one (1) application instances? Ideally, each additional application instance will contribute 100% of its individual capability, but this may not be the case as the cloud gets loaded with multiple application instances or other workloads (for public clouds). Scalability is reported as a unit-less number @ the number of valid application instances. The number is an aggregate of workloads metrics across all application instances running in a cloud normalized by workload metrics in a reference cloud platform. The reference platform metrics are an average of workload metrics taken across multiple cloud platforms during the benchmark development.

SUT

One or more cloud services under test. This includes all hardware, network, base software and management systems used for the cloud service.

Variability

The difference between different runs of the benchmark. Typically, variability arises due to factors such as multi-tenancy and time of day execution, and may be more pronounced on public clouds.

Whitebox Cloud

The SUT's exact engineering specifications including all hardware and software are known and under the control of the tester. This will typically be the case for private clouds.