Error:

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Power Profile set to Maximum Performance
• Thermal Configuration set to Maximum Cooling
• Turbo Mode:Enable

Storage Notes

• 2 x 600GB internal SAS drivers were connected to the LSI3108 controller,built as RAID1 and used for OS
• The SUT was connected to 3 x HUAWEI Dorado5100 via 3 x Quad port QLogic Corp ISP2714-based 16/32Gb Fibre Channel HBA.
• 3 x HUAWEI Dorado5100 holds 12 x 24 x 200 GB SSD drives. There are 12 RAID10 arrays, each consisting of 24 drives . Each array has 6 LUNs, and each LUN was partitioned for the VMs of 1 tile, though only 68 are used.

Virtualization Software Notes

• The following lines were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6 = 1
• net.ipv4.icmp_echo_ignore_broadcasts = 1
• net.ipv4.conf.all.rp_filter = 1
• fs.inotify.max_user_watches = 65536
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.netdev_max_backlog = 500000
• fs.aio-max-nr = 800000
• kernel.sched_min_granularity_ns = 10000000
• kernel.sched_wakeup_granularity_ns = 15000000
• kernel.sched_latency_ns = 20000000
• kernel.sched_migration_cost_ns = 215000
• kernel.numa_balancing = 0
• vm.swappiness = 0
• vm.dirty_ratio = 10
• vm.dirty_background_ratio = 5
• The following boot options were used:
• intel_iommu=on iommu=pt selinux=0 pci=realloc nohz=on highres=on hpet=disable audit=0 elevator=deadline clocksource=tsc tsc=reliable cgroup_disable=memory,cpu,cpuacct,blkio,hugetlb intel_idle.max_cstate=0 nmi_watchdog=0 console=tty0 console=ttyS0,115200
• The following services are disabled:
• atd, auditd, autofs, autoyast-initscripts, blk-availability, bmc-snmp-proxy, brltty, console-getty, console-shell, cron, debug-shell, dm-event, dmraid-activation, dnsmasq, exchange-bmc-os-info, gpm, grub2-once, ipmi, ipmievd, irqbalance, iscsid, iscsiuio, kdump-rebuild-initrd, kdump, kexec-load, klog, libvirt-guests, lunmask, lvm2-monitor, multipathd, nfs-blkmap, nfs-server, nfs, nfsserver, ntp-wait, ntpd, numad, openct, postfix, radvd, rpcbind, rsyncd, rtkit-daemon, saslauthd, serial-getty@, slpd, SuSEfirewall2, SuSEfirewall2_init, sysstat, systemd-bootchart, systemd-journal-upload, systemd-nspawn@, systemd-timesyncd, vsftpd, winbind, xinetd, ypbind
• The source code for the Intel kernel module ixgbe version 5.1.3 was download from the official web site, built with the default configuration options, and installed
• ixgbe module option used: max_vfs=63,63,63,63,63,63,63,63,63,63,63,63 InterruptThrottleRate=4000
• 6 Intel X520 configured for SUT-client ,web-infra and app-db communication.
• VMs are bound to host CPUs via their XML configuration in the following manner:
• All VMs of tiles 1-16,tile68 are bound to NUMA node 0
• All VMs of tiles 17-32,tile67 are bound to NUMA node 1
• All VMs of tiles 33-48,tile66 are bound to NUMA node 2
• All VMs of tiles 49-64,tile65 are bound to NUMA node 3
• VM configurations are as follows:
• All VMs are managed with libvirt and created using virsh define & virsh start
• All VMs are created with the following xml template:
• <domain type='kvm'>
• <name>$vm_name</name>
• <memory>$vm_memory</memory>
• <currentMemory>$vm_memory</currentMemory>
• <vcpu cpuset='$host-cpu-list'>$vm_cpu</vcpu>
• <os>
• <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <apic eoi='on'/>
• <pae/>
• <acpi/>
• </features>
• <cpu mode='custom' match='exact'>
• <model fallback='allow'>SandyBridge</model>
• <vendor>Intel</vendor>
• <topology sockets=$sockets cores=$cores threads=$threads/>
• </cpu>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>destroy</on_crash>
• <devices>
• <emulator>/usr/bin/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk' />
• <target dev='vda' bus='virtio'/>
• </disk>
• <hostdev mode='subsystem' type='pci' managed='yes'>
• <source>
• <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </hostdev>
• <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
• <listen type='address' address='0.0.0.0'/>
• </graphics>
• <memballoon model='none'>
• <alias name='balloon0'/>
• </memballoon>
• </devices>
• </domain>
• The following section is repeated for the second PCI device used in the app,db,web and infra VMs:
• <hostdev mode='subsystem' type='pci' managed='yes'>
• <source>
• <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </hostdev>
• The following section is repeated for the second disk used in the db,mail and infra VMs for their data:
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk' />
• <target dev='vdb' bus='virtio'/>
• </disk>
• App VMs are reniced -10.
• Db VMs are reniced -8.
• Web VMs are reniced -8.
• mail VMs are reniced 19.
• infra VMs are reniced 19.
• Batch VMs are reniced -10.

Web Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline cgroup_disable=memory,blkio
• The following services are disabled:
• abrtd atd auditd htcacheclean httpd ip6tables iptables irqbalance kdump netfs nfs nfslock nginx-debug ntpd ntpdate php-fpm postfix psacct quota_nld rdisc rhnsd rhsmcertd rngd rpcgssd rpcidmapd rpcsvcgssd saslauthd smartd
• boot file-system options: barrier=0,noatime
• root file-system options: barrier=0,noatime
• nfs file-system defaults options: noatime,actimeo=14400,nocto
• Contents of nginx configuration file /etc/nginx/nginx.conf
• user nginx;
• worker_processes 4;
• worker_cpu_affinity 00000001 00000001 00000010 00000010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request"$status" $body_bytes_sent ';
• access_log /var/log/access.log common gzip buffer=4m flush=1m;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• }
• Contents of nginx configuration file /etc/nginx/conf.d/default.conf
• server {
• listen 80 backlog=8192;
• server_name localhost;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ \.php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Contents of nginx configuration file /etc/nginx/conf.d/example_ssl.conf
• server {
• listen 443 ssl;
• server_name localhost;
• ssl on;
• ssl_certificate /etc/pki/tls/certs/localhost.crt;
• ssl_certificate_key /etc/pki/tls/private/localhost.key;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
• ssl_ciphers ALL:!kEDH!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ \.php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness = 0
• net.ipv4.tcp_fin_timeout=60
• net.ipv4.tcp_tw_recycle=1
• The following items were added to /etc/bashrc:
• ulimit -n 1048576
• ulimit -u 1000000
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=2000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 800000
• hard and soft nproc set to 102400
• The following lines were changed in /etc/php-fpm.d/www.conf
• listen = /dev/shm/php-cgi.sock
• listen.owner = nginx
• listen.group = nginx
• user = nginx
• group = nginx
• pm.max_children = 70

Application Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline cgroup_disable=memory,blkio
• The following services are disabled:
• abrtd atd auditd autofs cgred cups dnsmasq ip6tables iptables irqbalance kdump mcelogd netfs nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc rhnsd rhsmcertd rngd rpcgssd rpcidmapd rpcsvcgssd saslauthd smartd sssd winbind wpa_supplicant ypbind
• /etc/fstab changes:
• root file-system options: barrier=0
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• ixgbevf module option used: InterruptThrottleRate=2500
• /etc/security/limits.conf changes:
• * soft nofile 1048576
• * hard nofile 1048576
• * soft nproc 130172
• * hard nproc 130172
• spec soft nproc 130172
• spec hard nproc 130172
• * soft memlock 8388608
• * hard memlock 8388608
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• Update the JVM from 1.6.0 which is installed with the Websphere 8.5.5 to 1.7.0 which is download from IBM WebSite
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator.
• Heap size for AppSrv01 has been changed to 2048 MB.
• The following items were added to /etc/bashrc:
• ulimit -n 1048576
• ulimit -u 1000000

Mail Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline
• The following services are disabled:
• abrtd atd auditd ip6tables iptables irqbalance kdump netconsole netfs ntpd ntpdate postfix psacct quota_nld rdisc restorecond rhnsd rhsmcertd rngd saslauthd smartd
• boot file-system options: barrier=0 noatime
• root file-system options: barrier=0 noatime
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• Dovecot was compiled with default options and the following dovecot.conf was used:
• base_dir = /var/run/dovecot/
• ssl = no
• disable_plaintext_auth = no
• listen = *
• log_path = /var/log/dovecot
• mail_debug = no
• mmap_disable = yes
• mail_fsync = never
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 8192
• passdb {
• args = /etc/passwd.dovecot
• driver = passwd-file
• }
• protocols = imap
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• }
• userdb {
• driver = passwd
• }
• Following items were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6=1
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• The following items were added to /etc/bashrc:
• ulimit -n 1048576
• ulimit -u 1000000
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=1000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Database Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline
• The following services are disabled:
• abrtd atd auditd autofs cgred cups dnsmasq ip6tables iptables irqbalance kdump mcelogd netfs nfs nfslock ntpd ntpdate numad oddjobd portreserve postfix psacct quota_nld rdisc rhnsd rhsmcertd rngd rpcidmapd rpcsvcgssd saslauthd smartd sssd winbind wpa_supplicant ypbind
• /etc/fstab changes:
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• root file-system defaults options:barrier=0
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 610000
• db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5500000
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING
• alter tablespace C_parts_space NO FILE SYSTEM CACHING
• alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• alter tablespace M_parts_space NO FILE SYSTEM CACHING
• alter tablespace M_bom_space NO FILE SYSTEM CACHING
• alter tablespace M_wo_space NO FILE SYSTEM CACHING
• alter tablespace M_lo_space NO FILE SYSTEM CACHING
• alter tablespace M_inv_space NO FILE SYSTEM CACHING
• alter tablespace O_cust_space NO FILE SYSTEM CACHING
• alter tablespace O_ords_space NO FILE SYSTEM CACHING
• alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• alter tablespace O_item_space NO FILE SYSTEM CACHING
• alter tablespace S_comp_space NO FILE SYSTEM CACHING
• alter tablespace S_sc_space NO FILE SYSTEM CACHING
• alter tablespace S_po_space NO FILE SYSTEM CACHING
• alter tablespace S_site_space NO FILE SYSTEM CACHING
• alter tablespace S_supp_space NO FILE SYSTEM CACHING
• alter tablespace U_space NO FILE SYSTEM CACHING
• db2 update database config for specdb using LOGFILSIZ 65536
• db2 update db cfg for specdb using logprimary 54 logsecond 4
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• kernel.sem = 500 64000 400 512
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• vm.nr_hugepages=12288
• vm.hugetlb_shm_group=501
• vm.nr_overcommit_hugepages=512
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• db2inst1 soft data -1
• db2inst1 hard data -1
• db2inst1 soft nofile 65536
• db2inst1 hard nofile 65536
• db2inst1 soft fsize -1
• db2inst1 hard fsize -1
• db2inst1 soft nproc -1
• db2inst1 hard nproc -1
• * - nofile 1048576
• * soft memlock 8388608
• * hard memlock 8388608
• The tuned-0.2.19 package was installed and activated with virtual-guest profile

Infraserver VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline cgroup_disable=memory,blkio
• The following services are disabled:
• abrtd atd auditd htcacheclean ip6tables iptables irqbalance kdump netfs nfslock ntpd ntpdate postfix psacct quota_nld rdisc rhnsd rhsmcertd rngd rpcgssd rpcidmapd rpcsvcgssd saslauthd smartd
• boot file-system options: barrier=0,noatime
• root file-system options: barrier=0,noatime
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• httpd.conf changes:
• Timeout 20
• #MaxKeepAliveRequests 0
• #KeepAliveTimeout 60
• <IfModule prefork.c>
• StartServers 10
• MinSpareServers 10
• MaxSpareServers 10
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• <IfModule worker.c>
• StartServers 20
• MaxClients 4096
• MinSpareThreads 25
• MaxSpareThreads 75
• ThreadsPerChild 25
• MaxRequestsPerChild 640000
• Listen 81
• ServerName besim:80
• httpd.conf adds:
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgiIpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin">
• AllowOverride None
• Options +ExecCGI -Includes
• SetHandler fastcgi-script
• Order allow,deny
• Allow from all
• </Directory>
• Addhandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• The following httpd modules were loaded:
• file_cache mem_cache fastcgi
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness = 0
• The following items were added to /etc/bashrc:
• ulimit -n 1048576
• ulimit -u 1000000
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=3000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400
• NFS export options:
• rw,no_subtree_check,crossmnt,no_root_squash

Batch Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline cgroup_disable=memory,blkio
• The following services are disabled:
• abrt-ccpp abrt-oops abrtd atd auditd ip6tables iptables irqbalance kdump mdmonitor messagebus netfs ntpd ntpdate postfix psacct quota_nld rdisc rhnsd rhsmcertd rngd saslauthd smartd
• boot file-system options: barrier=0
• root file-system options: barrier=0
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following items were added to /etc/bashrc:
• ulimit -n 1048576
• ulimit -u 1000000
• The tuned-0.2.19 package was installed and activated with virtual-guest profile
• ixgbevf module option used: InterruptThrottleRate=7000
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Client Driver Notes

• The client systems VMs using Red Hat Enterprise Linux 6.4.
• One virtual client were used for each tile.
• One virtual client per tile ran all the workloads.
• Each prime client VM has 5 VCPUs, 10240 MB memory, and a 20 GB disk,1 ixgbevf device and 1 virtio_blk device are used.

Other Notes

• A Quidway S9303 36-port switch was used to connect client systems to SUT. The 10GbE ports were attached to a dual port 10 GbE SFP+ expansion module.