Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• ILOM settings used:
• ILOM web interface: remove the limits at menu path
• Power Management / Limit

Storage Notes

• OS installed on internal 4x 600GB disks. LDOM1 and LDOM2 installed on raid-1 disk pairs.
• All Oracle Server X5-2L running Solaris 11.3 with following tunings
• - Oracle Server X5-2L BIOS - disable uncore frequency scaling
• - Solaris 11.3 /etc/system - set idle_cpu_no_deep_c=1
• - Solaris 11.3 /kernel/drv/qlt.conf -
• - bucketcnt2k=32768; bucketcnt8k=8192; bucketcnt64k=4096;
• - bucketcnt128k=16384; bucketcnt256k=128
• 1x Oracle Server X5-2L (2x 2.4GHz, 32GB mem, 4x 1600GB NVMe drives, 4x 400GB SSDs) configured with zone root pools and JMS log pools.
• - 4x 1600GB NVMe drives used in 2 Solaris Volume Manager (SVM) RAID1 mirrors, interlace=32b; Mirrors provided 10x 290GB SVM soft partitions, 5 exported on port A, 5 exported on port B used for the zone root pools
• - 4x 400GB SSDs used in 2 Solaris Volume Manager (SVM) RAID1 mirrors, interlace=32b; Mirrors provided 10x 20GB SVM soft partitions, 5 exported on port A, 5 exported on port B used for the JMS log pools
• 1x Oracle Server X5-2L (2x 2.4GHz, 32GB mem, 4x 1600GB NVMe drives) configured with web target file set in Infra zones and the web and infra log pools.
• - 4x 1600GB NVMe drives used in 2 Solaris Volume Manager (SVM) RAID1 mirrors, interlace=32b
• Mirrors provided 20x 40GB SVM soft partitions 10 exported on port A, 10 exported on port B used for the web target file set pools in Infra zones
• Mirrors provided 40x 20GB SVM soft partitions 20 exported on port A, 20 exported on port B used for the Web and Infra log pools
• 1x Oracle Server X5-2L (2x 2.4GHz, 32GB mem, 4x 1600GB NVMe drives) configured with mail store, and mail logs pools
• - 4x 1600GB NVMe drives used in 2 Solaris Volume Manager (SVM) RAID1 mirrors, interlace=32b
• Mirrors provided 18x 40GB SVM soft partitions 9 exported on port A, 9 exported on port B used for the mail store pools
• Mirrors provided 18x 20GB SVM soft partitions 9 exported on port A, 9 exported on port B used for the mail log pools
• 1x Oracle Server X5-2L (2x 2.4GHz, 32GB mem, 4x 1600GB NVMe drives) configured with database zone storage
• - 4x 1600GB NVMe drives used in 2 Solaris Volume Manager (SVM) RAID1 mirrors, interlace=32b
• Mirrors provided 36x 34GB SVM soft partitions 18 exported on port A, 18 exported on port B used in the database zones
• Mirrors provided 36x 32GB SVM soft partitions metadevices 9 exported on port A, 9 exported on port B used in the database zones

Virtualization Software Notes

• 32 cores, 478GB mem from processor 0 ran the LDOM1.
• 32 cores, 478GB mem from processor 1 ran the LDOM2.
• LDOM1 held VMs for tiles 1-4, 9-12, 17-20, 25-28
• LDOM2 held VMs for tiles 5-8, 13-16, 21-24, 29-32
• Tuning for LDOM1 and LDOM2:
• Oracle Solaris 11.3
• set autoup = 345600
• set maxphys=1048576
• set plat_disable_mempm=1
• set lpg_alloc_prefer=1
• set lgrp_mem_pset_aware=1
• set rlim_fd_max=100000
• set rlim_fd_cur=100000
• set zfs:zfs_arc_max=274877906944
• stop intrd
• Network Tuning:
• /etc/rc2.d/S99spec
• ndd -set /dev/tcp tcp_conn_req_max_q 819200
• ndd -set /dev/tcp tcp_conn_req_max_q0 819200
• ndd -set /dev/tcp tcp_xmit_hiwat 524288
• ndd -set /dev/tcp tcp_recv_hiwat 524288
• ndd -set /dev/tcp tcp_smallest_anon_port 2048
• ndd -set /dev/tcp tcp_naglim_def 1
• ndd -set /dev/tcp tcp_max_buf 4194304
• ndd -set /dev/tcp tcp_cwnd_max 2097152
• 4 virtual switches (etherstubs) created for inter-zone communications
• vsw0 (odd tiles) and vsw1 (even tiles) for non-benchmark inter-VM communication
• vsw2 for web/besim communication
• vsw3 for japp/DB communication
• 8x 10 Gbit NIC port (2 per PCI card) are virtualized and used for client/VM communication.
• Tuning in /kernel/drv/ixgbe.conf:
• LDOM1 ixgbe[2/3/4/5]: intr_throttling=4000
• LDOM2 ixgbe[0/1/2/3]: intr_throttling=4000
• LDOM1's ixgbe2, ixgbe3, ixgbe4, ixgbe5 handles the tiles on LDOM1
• LDOM2's ixgbe0, ixgbe1, ixgbe2, ixgbe3 handles the tiles on LDOM2
• LDOM1 Configuration Notes:
• All zones in the LDOM share the available 478 GB of memory
• psrset used to create
• - pset 0 - CPU 0-39, 192-199, 233-239 - OS
• - pset 1 - CPU 40-191 - bound web/infra/japp/mail VMs
• - pset 2 - CPU 200-232 - db VMs
• - pset 3 - CPU 240-255 - bound batch
• Interrupts to CPU binding:
• - ixgbe2/3/4/5 to 0-15
• - ixgbe0/1 - not used, not bound
• - emlxs0 to 17, emlxs1 to 18, emlxs2 to 19, emlxs3 to 20
• LDOM2 Configuration Notes:
• All zones in the LDOM share the available 478 GB of memory
• psrset used to create
• - pset 0 - CPU 0-39, 192-199, 233-239 - OS
• - pset 1 - CPU 40-191 - bound web/infra/japp/mail VMs
• - pset 2 - CPU 200-232 - bound db
• - pset 3 - CPU 240-255 - bound batch
• Interrupts to CPU binding:
• - ixgbe0/1/2/3 to 0-15
• - ixgbe4/5 - not used, not bound
• - emlxs0 to to 17, emlxs1 to 18, emlxs2 to 19, emlxs3 to 20
• All VMs are managed with zonadm and created using zone cfg utility
• All VMs are created using the following XML template with VM specific exceptions noted below:
• <?xml version="1.0" encoding="UTF-8"?>
• <!DOCTYPE zone PUBLIC "-//Sun Microsystems Inc//DTD Zones//EN" "file:///usr/share/lib/xml/dtd/zonecfg.dtd.1">
• <zone name="<"zone name> zonepath="<zone root dir>" autoboot="false" brand="solaris" ip-type="exclusive" limitpriv="default,dtrace_proc,dtrace_user">
• <network physical="<administrative network (not used for BM)"> configure-allowed-address="true"/>
• DB VM modifications
• <network physical="<vnic to japp VM>" configure-allowed-address="true"/>
• <network physical="<ixgbe-vnic to client>" configure-allowed-address="true"/>
• <device match="<block data>"/>
• <device match="<raw date>"/>
• <device match="<block data>"/>
• <device match="<raw date>"/>
• <device match="<block redo1>"/>
• <device match="<raw redo1>"/>
• <device match="<block redo2>"/>
• <device match="<raw redo2>"/>
• <device match="<block DB_Backup>"/>
• <device match="<raw DB_Backup>"/>
• <device match="<block DB_Backup>"/>
• <device match="<raw DB_Backup>"/>
• <device match="<block redo1 backup>"/>
• <device match="<raw redo1 backup>"/>
• <device match="<block redo2 backup>"/>
• <device match="<raw redo2 backup>"/>
• batch VM modifications
• <network physical="<ixgbe-vnic to client>" configure-allowed-address="true"/>
• Infra VM modifications
• <network physical="<ixgbe-vnic to client>" configure-allowed-address="true"/>
• <network physical="<vnic to web VM>" configure-allowed-address="true"/>
• <dataset name="<target file set>" alias="data"/>
• <dataset name="<infra log>" alias="http_logs"/>
• jApp VM modifications
• <network physical="ixgbe-vnic to client" configure-allowed-address="true"/>
• <network physical="vnic to db VM" configure-allowed-address="true"/>
• <dataset name="<jms log>" alias="jms"/>
• Mail VM modifications
• <network physical="ixgbe-vnic to client" configure-allowed-address="true"/>
• <dataset name="mail store disk" alias="data"/>
• <dataset name="mail log" alias="mail_logs"/>
• Web VM modifications
• <network physical="ixgbe-vnic to client" configure-allowed-address="true"/>
• <network physical="vnic to infra VM" configure-allowed-address="true"/>
• <dataset name="web server log" alias="http_logs"/>

Web Server VM Notes

• MPSSHEAP=256M
• iPlanet processes started in the FX class using the priocntl -c FX -e
• magnus.conf file changes:
• remove - Init fn="load-modules" shlib="libj2eeplugin.so
• add - Init fn="load-modules" shlib="libfastcgi.so
• obj.conf file changes:
• remove the following lines:
• NameTrans fn="ntrans-j2ee" name="j2ee
• PathCheck fn="find-index-j2ee
• PathCheck fn="find-index" index-names="index.html,home.html,index.jsp
• ObjectType fn="type-j2ee
• Error fn="error-j2ee
• add the following lines:
• NameTrans fn="pfx2dir" from="/support" dir="/export/home/docs/support" name="es-internal
• PathCheck fn="find-index" index-names="index.html,home.html,index.jsp,index.php
• <If -f $path>
• Service type="magnus-internal/php
• fn="responder-fastcgi
• app-path="/usr/php/5.3/bin/php-cgi
• bind-path="localhost:3101
• app-env="PHPRC=/opt/oracle/webserver7/plugins/fastcgi
• app-env="PHP_INI_SCANDIR=< < /etc/php/5.3/conf.d > >
• app-env="PHP_FCGI_CHILDREN=12
• app-env="PHP_FCGI_MAX_REQUESTS=2000
• app-env="FCGI_WEB_SERVER_ADDRS=127.0.0.1
• bucket="php-bucket >
• </If>
• <Else>
• Service type="magnus-internal/php" fn="set-variable" error="404
• </Else>
• iPlanet configured to use T7 HW Crypto Accelerator: Cert name is Sun Metaslot:T4-cert
• server.xml file changes:
• log-file: /http_logs/errors
• log-stdout: false
• log-stderr: false
• log-to-console: false
• log-level: failure
• platform:64
• keep-alive
• threads: 12
• max-connections: 26250
• timeout: 120
• poll-interval: 0.05
• thread-pool
• queue-size: 26250
• min-threads: 104
• max-threads: 104
• stack-size: 131072
• file-cache
• max-age: 3600
• max-entries: 10000
• max-open-files: 10000
• sendfile: true
• sendfile-size: 262144
• replacement: false
• max-heap-file-size: 600000
• max-heap-space: 100000000
• max-mmap-file-size: 100000
• cache-content: true
• http
• etag: false
• access-log: /http_logs/access
• http-listener-1
• port: 443
• acceptor-threads: 8
• listen-queue-size: 32768
• server-cert-nickname: Sun Metaslot:T4-cert
• ssl3-tls-ciphers: SSL_RSA_WITH_RC4_128_MD5
• http-listener-2
• port: 80
• acceptor-threads: 2
• listen-queue-size: 8192
• virtual-server
• http-listener-name: http-listener-1
• http-listener-name: http-listener-2
• document-root: /export/home/docs

Application Server VM Notes

• application server:
• Java processes started in FX class using the priocntl -c FX -e
• JAVA_OPTIONS= -server -Xms1700m -Xmx1700m -Xmn1000m -Xss128k -XX:+AggressiveHeap -XX:+UseParallelGC -XX:ParallelGCThreads=8 -XX:PermSize=128m -XX:LargePageSizeInBytes=4m -XX:+UseParallelOldGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
• WL_OPTIONS: -Dweblogic.management.discover=false -Dweblogic.SocketReaders=2 -Dweblogic.StuckThreadMaxTime=900 -Dweblogic.diagnostics.debug.DebugLogger.DISABLED=true -Dweblogic.management.allowClearTextPasswords=true
• emulator
• JAVA_OPTIONS= -server -Xms1700m -Xmx1700m -Xmn1g -Xss128k -XX:+AggressiveHeap -XX:+UseParallelGC -XX:ParallelGCThreads=16 -XX:LargePageSizeInBytes=4m -XX:MaxTenuringThreshold=3 -XX:SurvivorRatio=20 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:-TraceClassUnloading -XX:PermSize=96m
• WL_OPTIONS: -Dweblogic.management.discover=false -Dweblogic.SocketReaders=4 -Dsun.net.inetaddr.ttl=0 -Dnetworkaddress.cache.ttl=0

Mail Server VM Notes

• ulimit -n 65536
• Dovecot processes started in the FX class using the priocntl -c FX -e
• Dovecot was compiled with default options. The dovecot.conf was used:
• dict {
• }
• #!include conf.d/*.conf
• base_dir = /export/home/dovecot/
• default_login_user = dovecot
• disable_plaintext_auth = no
• log_path = /mail_logs/dovecot.log
• info_log_path = /mail_logs/debug_info_dovecot.log
• mail_debug = yes
• auth_verbose=yes
• auth_debug=yes
• auth_debug_passwords=yes
• verbose_ssl=yes
• mail_location = dbox:/export/home/vmail/%u
• mbox_write_locks = fcntl
• mail_fsync = optimized
• passdb {
• args = scheme=plain username_format=%u /export/home/dovecot/etc/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = postmaster@example.com
• protocols = imap
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• userdb {
• driver = passwd
• }
• Filesystems for mail store (/data) and logs (/mail_logs) have a ZFS recordsize of 16K.

Database Server VM Notes

• ASM and DB processes started in FX class using the priocntl -c FX -e
• db_name = specdb
• control_files = +DBDATA/cntrlspecdb
• compatible = 11.2.0.3.0
• cpu_count = 4
• db_cache_size = 1024M
• db_8k_cache_size = 400M
• db_4k_cache_size = 400M
• db_file_multiblock_read_count = 128
• db_block_size = 2048
• db_files = 256
• db_file_multiblock_read_count = 128
• db_writer_processes = 8
• dml_locks = 300
• filesystemio_options = setall
• log_checkpoint_interval = 0
• log_checkpoints_to_alert = TRUE
• processes = 500
• open_cursors = 2000
• shared_pool_size = 400M
• sessions = 400
• transactions = 400
• transactions_per_rollback_segment = 1
• parallel_max_servers = 100
• undo_management = AUTO
• undo_retention = 5
• undo_tablespace = undo_ts
• statistics_level = BASIC
• timed_statistics = FALSE

Infraserver VM Notes

• iPlanet processes started in the FX class using the priocntl -c FX -e
• magnus.conf file changes:
• remove - Init fn="load-modules" shlib="libj2eeplugin.so
• add -
• Init fn="load-modules" shlib=/export/home/Besim_nsapi/Besim_SPARC_solaris/SPARC_besim_nsapi.so" funcs="service_besim,init_besim" shlib_flags="(global|now)
• Init fn="init_besim" LateInit="no
• MaxProcs 2
• obj.conf file changes:
• remove -
• AuthTrans fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true'
• NameTrans fn="ntrans-j2ee" name="j2ee
• NameTrans fn="pfx2dir" from="/mc-icons" dir="/opt/oracle/webserver7/lib/icons" name="es-internal
• PathCheck fn="uri-clean
• PathCheck fn="check-acl" acl="default
• PathCheck fn="find-pathinfo
• PathCheck fn="find-index-j2ee
• PathCheck fn="find-index" index-names="index.html,home.html,index.jsp
• ObjectType fn="type-j2ee
• ObjectType fn="type-by-extension
• ObjectType fn="force-type" type="text/plain
• Service method="(GET|HEAD)" type="magnus-internal/directory" fn="index-common
• Service method="(GET|HEAD|POST)" type="*~magnus-internal/*" fn="send-file
• Service method="TRACE" fn="service-trace
• Error fn="error-j2ee
• add -
• NameTrans fn="assign-name" from="/BeSim/specweb2005_besim" name="besim" nostat="/BeSim/specweb2005_besim" find-pathinfo-forward=true
• <Object name="besim">
• ObjectType fn="force-type" type="magnus-internal/besim
• Service fn="service_besim" method="(GET|POST)" type="magnus-internal/besim
• </Object>
• server.xml file changes:
• log-file: /http_logs/errors
• log-stdout: false
• log-to-console: false
• log-level: failure
• platform:32
• user: webservd
• keep-alive
• threads: 4
• max-connections: 8192
• timeout: 3600
• thread-pool
• queue-size: 8192
• min-threads: 65
• max-threads: 65
• stack-size: 131072
• stats:
• enabled: false
• qos
• enabled: false
• ssl-session-cache
• enabled: false
• access-log: /http_logs/access
• http-listener
• port: 81
• listen-queue-size: 8192

Batch Server VM Notes

• Batch process started in TS class using the priocntl -e -c TS -m -30 -p -30

Client Driver Notes

• set sq_max_size=0
• set rlim_fd_max=32768
• set rlim_fd_cur=32768
• set ddi_msix_alloc_limit=8
• set ip:ip_soft_rings_10gig_cnt=16
• set segkpsize=524288
• ndd -set /dev/tcp tcp_conn_req_max_q 40000
• ndd -set /dev/tcp tcp_conn_req_max_q0 40000
• ndd -set /dev/tcp tcp_xmit_hiwat 524288
• ndd -set /dev/tcp tcp_recv_hiwat 524288
• ndd -set /dev/tcp tcp_naglim_def 1
• ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 85000
• ndd -set /dev/tcp tcp_smallest_anon_port 1200
• ndd -set /dev/tcp tcp_max_buf 8388608
• ndd -set /dev/tcp tcp_cwnd_max 4194304
• Disable SunPKCS11 security provider for Java
• Reorder remaining security providers by decrementing provider number by 1
• Enabled SSLv3 algorithm in java.security for benchmark use.

Other Notes

• One Summit X650-24t 24 port 10 GbE Switch
• One 48 port Brocade 6510 Fiber Switche.

Performance Details: