Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Power Profile set to Maximum Performance
• Thermal Configuration set to Maximum Cooling
• Turbo Mode:Enable

Storage Notes

• All storage accessed with 4 x 8 Gb Fibre Channel links. And the SUT was accessed via for fibre channel controllers
• 1 x OceanStor S5900 unit was used with 7 S5900 disk enclousures attached to the S5900 via 1 x 8 Gb drive port. Each S5900 holds 24 x 300 Gb SAS drives. Total drive count is 168. There are 7 RAID10 arrays, each consisting of 24 drives (168 drives used). Each array has only one LUN. There are 7tiles system image on this S5900 (7 LUNs used).
• 3 x HUASY Dorado2100 units were used via 1 x 8 Gb drive port each. Each HUASY Dorado2100 holds 24 x 100 GB SSD drives. There are 3 RAID10 arrays, each consisting of 24 drives (72 drives used). One array has 2 LUNs for two dbserver's datastatore,the other two array has 7 LUNs each for 7 tiles mailserver data and infraserver data.
• The Server ProLiant DL380p Gen8 of SUT holds 3 x 300 GB SAS drives. There is 1 RAID5 array, consisting of 3 drives. This array was used as local disk for installing Virtualization Platform. And the boot device is on the local disk /dev/sda.

Virtualization Software Notes

• The following lines were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6 = 1
• net.ipv4.icmp_echo_ignore_broadcasts = 1
• net.ipv4.conf.all.rp_filter = 1
• fs.inotify.max_user_watches = 65536
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.netdev_max_backlog = 500000
• fs.aio-max-nr = 70656
• vm.swappiness = 10
• vm.dirty_ratio = 10
• vm.dirty_background_ratio = 5
• The following boot options were used:
• showopts hpet=disable clocksource=tsc tsc=reliable nohz=on highres=off elevator=deadline selinux=0 audit=0 intel_iommu=on iommu=pt cgroup_disable=memory,blkio nmi_watchdog=0
• The following services are disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, dnsmasq, ebtables, ip6tables, iptables, irqbalance, iscsi, iscsid, ksm, ksmtuned, mdmonitor, netconsole, ntpdate, numad, oddjobd, postfix, psacct, quota_nld, radvd, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcsvcgssd, saslauthd, smartd, sssd, vncserver, winbind, ypbind
• After boot up, the following options were used:
• echo 160000000 > /proc/sys/kernel/sched_latency_ns
• echo 80000000 > /proc/sys/kernel/sched_min_granularity_ns
• echo 150000000 > /proc/sys/kernel/sched_wakeup_granularity_ns
• ixgbe module option used: InterruptThrottleRate=4000
• 1 Intel X540 for internal communication between WEBserver & INFRAserver, 2 Intel X520 configured for SUT-client and APP-DB communication. A maximum of 378 virtual functions were configured and 60 virtual functions were used for the Web, App, DB, Mail Infra and Batch VMs.
• VMs are bound to host CPUs via their XML configuration in the following manner:
• The 1st dbserver and 2nd dbserver were bound to NUMA node 0 and prefer to allocate memory from node 0
• All VMs of tiles 1,2,3,4 (except the 1st dbserver which are bound to NUMA node 0) are bound to NUMA node 1 prefer to allocate memory from node 1
• All VMs of tiles 5,6,7 (except all of the mailservers are bound to NUMA node 1) are bound to NUMA node 0 prefer to allocate memory from node 0
• Web, Mail, and Infra VMs have a cputune shares value of 1024
• DB and APP VMs have a cputune shares value of 16384
• Batch VMs have a cputune shares value of 128
• VM configurations are as follows:
• All VMs are managed with libvirt and created using virsh define & virsh start
• All VMs are created with the following xml template, exceptions noted below template. (Each VM's xml template is at System_Under_Test/vmconfig directory):
• <domain type='kvm'>
• <name>$vm_name</name>
• <memory>$vm_memory</memory>
• <numatune>
• <memory mode='strict' nodeset='$vm_node'/>
• </numatune>
• <currentMemory>$vm_memory</currentMemory>
• <vcpu cpuset='$host-cpu-list'>$vm_cpu</vcpu>
• <cpu>
• <model>SandyBridge</model>
• <feature policy='force' name='x2apic'/>
• </cpu>
• <os>
• <type arch='x86_64' machine='pc'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <pae/>
• <apic eoi='on'/>
• <acpi/>
• </features>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>destroy</on_crash>
• <devices>
• <emulator>/usr/libexec/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk' />
• <target dev='vda' bus='virtio'/>
• </disk>
• <hostdev mode='subsystem' type='pci' managed='no'>
• <source>
• <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </hostdev>
• <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
• <listen type='address' address='0.0.0.0'/>
• </graphics>
• <memballoon model='none'>
• <alias name='balloon0'/>
• </memballoon>
• </devices>
• </domain>
• App Web Batch VMs do not use qemu for disk emulator, and therefore have the following declaration instead of 'disk type' block above:
• <controller type='scsi' model='vhost-scsi' index='0'>
• <source wwpn='naa.$vm_scsi' event_idx='on'/>
• <driver queue='$queue_num'/>
• </controller>
• Mail VMs do not use qemu for storage disk emulator, and therefore have the following declaration instead of 'disk type' block above:
• <disk type='volume' device='disk'>
• <driver name='qemu' type='raw'/>
• <source pool='$pool' volume='$unit_ID'/>
• <target dev='sda' bus='scsi'/>
• </disk>

Web Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0,noatime
• nfs file-system defaults options: noatime,actimeo=14400,nocto
• MTU=9000 for eth1 (internal network).
• The MTU is set after all VMs are started through the following command: ifconfig eth1 mtu 9000 (the script setmtu.sh runs this command and the script is at System_Under_Test directory).
• httpd.conf changes:
• MaxKeepAliveRequests 0
• Timeout 60
• KeepAliveTimeout 60
• StartServers 10
• MinSpareServers 25
• MaxSpareServers 75
• ServerLimit 4400
• MaxClients 4400
• MaxRequestsPerChild 1024
• CustomLog logs/access_log common
• Include conf.d/*.conf
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness = 0
• net.ipv4.tcp_fin_timeout=60
• net.ipv4.tcp_tw_recycle=1
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Application Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0
• Glassfish domain.xml adds:
• To jdbc-connection-pool line parameter 'max-pool-size = 512' was set
• To request-processing line parameter 'initial-thread-count = 90' was set
• To ejb-container line, parameter 'max-pool-size= 512' was set
• To mdb-container line, parameter 'max-pool-size = 512' was set
• jvm-options '-Xmn1600m' was set
• jvm-options '-XX:+DisableExplicitGC' was set
• jvm-options '-XX:+UseParallelOldGC' was set
• jvm-options '-XX:+UseLargePages' was set
• jvm-options '-XX:-UseBiasedLocking' was set
• jvm-options '-XX:CompileThreshold=1000' was set
• jvm-options '-XX:ParallelGCThreads=4' was set
• jvm-options '-XX:+AggressiveOpts' was set
• jvm-options '-XX:+UseCompressedOops' was set
• jvm-options '-XX:+UseFastAccessorMethods' was set
• jvm-options '-XX:+UseStringCache' was set
• jvm-options '-XX:+OptimizeStringConcat' was set
• jvm-options '-XX:LargePageSizeInBytes=2m' was set
• domain.xml changes:
• jvm-options '-Xmx4800m' was set
• jvm-options '-Xms4800m' was set
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• nvm.swappiness=0
• The following items were added to /etc/bashrc:
• ulimit -n 500000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft memlock set to 3072000
• hard and soft nofile set to 130172
• hard and soft nproc set to 130172

Mail Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd svnserve winbind ypbind
• /etc/fstab changes:
• root file-system options: barrier=0
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• Dovecot was compiled with default options and the following dovecot.conf was used:
• base_dir = /var/run/dovecot/
• ssl = no
• disable_plaintext_auth = no
• listen = *
• log_path = /var/log/dovecot
• mail_debug = no
• mmap_disable = yes
• mail_fsync = never
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 8192
• passdb {
• args = /etc/passwd.dovecot
• driver = passwd-file
• }
• protocols = imap
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• }
• userdb {
• driver = passwd
• }
• Following items were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6=1
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• The following items were added to /etc/bashrc:
• ulimit -n 100000
• ulimit -u 10240
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Database Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrt-ccpp abrtd atd auditd autofs certmonger cgconfig cgred crond cups haldaemon ip6tables iptables irqbalance kdump mcelogd messagebus netconsole nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd saslauthd smartd sssd svnserve winbind ypbind
• /etc/fstab changes:
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• Database tuning options:
• large-pages
• lower_case_table_name = 1
• autocommit = 0
• datadir = /dbstore/mysql
• max_connections = 1500
• max_connect_errors = 6000
• innodb_buffer_pool_size = 8192M
• innodb_buffer_pool_instances = 1
• innodb_flush_method = O_DIRECT
• innodb_log_buffer_size= 1024M
• innodb_log_file_size= 1024M
• innodb_log_files_in_group = 3
• innodb_lock_wait_timeout = -1
• innodb_additional_mem_pool_size = 16M
• table_open_cache = 30000
• table_open_cache_instances = 16
• sort_buffer_size = 8M
• read_buffer_size = 8M
• read_rnd_buffer_size = 512M
• key_buffer_size = 32M
• back_log = 1560
• thread_cache_size = 128
• tmp_table_size = 512M
• max_allowed_packet = 128M
• skip-name-resolve = 1
• innodb_checksums = 0
• innodb_doublewrite = 0
• innodb_support_xa = 0
• innodb_thread_concurrency = 0
• innodb_flush_log_at_trx_commit = 0
• innodb_max_dirty_pages_pct= 50
• innodb_use_native_aio=1
• innodb_stats_persistent = 1
• innodb_adaptive_flushing = 1
• innodb_flush_neighbors = 0
• innodb_read_io_threads = 8
• innodb_write_io_threads = 8
• innodb_io_capacity = 40000
• innodb_purge_threads = 1
• innodb_adaptive_hash_index = 1
• innodb_monitor_enable = '%'
• performance_schema = OFF
• Following items were added to /etc/sysctl.conf:
• net.ipv4.icmp_echo_ignore_broadcasts = 1
• net.ipv4.conf.all.rp_filter = 1
• fs.inotify.max_user_watches = 65536
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.netdev_max_backlog = 500000
• fs.aio-max-nr = 70656
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• kernel.sem = 500 64000 400 512
• net.ipv4.neigh.default.gc_thresh1 = 4096
• net.ipv4.neigh.default.gc_thresh2 = 8192
• net.ipv4.neigh.default.gc_thresh3 = 8192
• net.ipv4.neigh.default.base_reachable_time = 86400
• net.ipv4.neigh.default.gc_stale_time = 86400
• net.ipv4.tcp_tw_reuse = 1
• net.ipv6.conf.all.disable_ipv6=1
• vm.nr_hugepages = 4600
• vm.hugetlb_shm_group = 498
• The following items were added to /etc/bashrc:
• ulimit -n 100000
• ulimit -u 10240
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Infraserver VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0,noatime
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• MTU=9000 for eth1 (internal network).
• The MTU is set after all VMs are started through the following command: ifconfig eth1 mtu 9000 (the script setmtu.sh runs this command and the script is at System_Under_Test directory).
• httpd.conf changes:
• MaxKeepAliveRequests 0
• Timeout 20
• KeepAliveTimeout 60
• StartServers 20
• MinSpareServers 20
• MaxSpareServers 20
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness = 0
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Batch Server VM Notes

• The following boot options were used:
• hpet=disable highres=off cgroup_disable=memory,blkio nohz=on selinux=0 audit=0 watchdog=0 clocksource=tsc tsc=realiable elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs avahi-daemon cgconfig httpd iptables ip6tables irqbalance iscsi iscsid ksm ksmtuned NetworkManager postfix rhnsd rpcgssd rpcidmapd sysstat
• /etc/fstab changes:
• root file-system options: barrier=0
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Client Driver Notes

• The client systems VMs using Red Hat Enterprise Linux 6.4.
• Two virtual clients were used for each tile,One virtual client per tile ran the webserver workload and the other virtual client per tile ran the other workloads.
• The first virtual client is the prime controller.
• Each prime client VM has 10 VCPUs, 10240 MB memory, and a 20 GB disk,1 ixgbevf device and 1 virtio_blk device are used.

Other Notes

• A Quidway S9303 36-port switch was used to connect client systems to SUT. The 10GbE ports were attached to a dual port 10 GbE SFP+ expansion module.

Performance Details: