x86 Open64 Compiler Suite SPEC CPU2006 Flag Description

Compilers: x86 Open64 Compiler Suite


Sections

Selecting one of the following will take you directly to that section:


Optimization Flags


Portability Flags


Compiler Flags


Other Flags


Commands and Options Used to Submit Benchmark Runs

Using numactl to bind processes and memory to cores

For multi-copy runs or single copy runs on systems with multiple sockets, it is advantageous to bind a process to a particular core. Otherwise, the OS may arbitrarily move your process from one core to another. This can effect performance. To help, SPEC allows the use of a "submit" command where users can specify a utility to use to bind processes. We have found the utility 'numactl' to be the best choice.

numactl runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for a command and inherited by all of its children. The numactl flag "--physcpubind" specifies which core(s) to bind the process. "-l" instructs numactl to keep a process memory on the local node while "-m" specifies which node(s) to place a process memory. For full details on using numactl, please refer to your Linux documentation, 'man numactl'

Note that some versions of numactl, particularly the version found on SLES 10, we have found that the utility incorrectly interprets application arguments as it's own. For example, with the command "numactl --physcpubind=0 -l a.out -m a", numactl will interpret a.out's "-m" option as it's own "-m" option. To work around this problem, a user can put the command to be run in a shell script and then run the shell script using numactl. For example: "echo 'a.out -m a' > run.sh ; numactl --physcpubind=0 bash run.sh"

numactl is also used to invoke runspec so that mememory usage is spread evenly among NUMA nodes. This is accomplished as follows: runspec_command="numactl --interleave=all runspec"


Shell, Environment, and Other Software Settings

Linux Huge Page settings

In order to take full advantage of using x86 Open64's huge page runtime library, your system must be configured to use huge pages. It is safe to run binaries compiled with "-HP" on systems not configured to use huge pages, however, you will not benefit from the performance improvements huge pages offer. To configure your system for huge pages perform the following steps:

Note that further information about huge pages may be found in your Linux documentation file: /usr/src/linux/Documentation/vm/hugetlbpage.txt

HUGETLB_LIMIT

For the x86 Open64 compiler, the maximum number of huge pages an application is allowed to use can be set at run time via the environment variable HUGETLB_LIMIT. If not set, then the process may use all available huge pages when compiled with "-HP (or -HUGEPAGE)" or a maximum of n pages where the value of n is set via the compile time flag "-HP:limit=n".

Transparent Huge Pages (THP)

THP is an abstraction layer that automates most aspects of creating, managing, and using huge pages. THP is designed to hides much of the complexity in using huge pages from system administrators and developers, as normal huge pages must be assigned at boot time, can be difficult to manage manually, and often require significant changes to code in order to be used effectively.

Set transparent_hugepage boot parameter

In the file /boot/grub/menu.lst, add the boot parameter "transparent_hugepage=never" to the OS you plan to select during boot, to instruct it to disable Transparent Huge Pages (THP). A reboot is required for this setting to take effect.

Set Ubuntu power governor to performance

To produce the best performance on Ubuntu, the system power governor must be set to performance as follows:

ulimit -s <n>

Sets the stack size to n kbytes, or unlimited to allow the stack size to grow without limit.

ulimit -l <n>

Sets the maximum size of memory that may be locked into physical memory.

dirty_ratio

Sets the percentage limit of system memory that can hold dirty cache data until it is written out via pdflush.

swappiness

The swappiness value can range from 1 to 100. A value of 100 will cause the kernel to swap out inactive processes frequently in favor of file system performance, resulting in large disk cache sizes. A value of 1 tells the kernel to only swap processes to disk if absolutely necessary.

zone_reclaim_mode

When zone_reclaim_mode is set to 0, the kernel will allocate memory from a remote node, rather than try to reclaim memory from the local node. A value of 1 will cause the page allocator to reclaim local page caches that are not currently used before allocating remote node memory.

sync, drop_caches

Used in conjunction, the two commands, sync and drop_caches, free disk cache memory for other uses. sync writes dirty pages to disk, while drop_caches reclaims clean disk cache pages.

OMP_NUM_THREADS

Sets the maximum number of OpenMP parallel threads auto-parallelized (-apo) applications may use.

O64_OMP_AFFINITY_MAP

Specifies the thread-CPU relationship when the operating system's affinity mechanism is used to assign OpenMP threads to CPUs.

O64_OMP_SPIN_USER_LOCK

Specifies whether or not to use the user-level spin mechanism for OpenMP locks. If the variable is set to TRUE then user-level spin mechanisms are used. If the variable is set to FALSE then pthread mutexes are used. The default if the variable is not set is the same as FALSE.

powersave -f (on SuSE)

Makes the powersave daemon set the CPUs to the highest supported frequency.

/etc/init.d/cpuspeed stop (on Red Hat)

Disables the cpu frequency scaling program in order to set the CPUs to the highest supported frequency.

LD_LIBRARY_PATH

An environment variable set to include the x86 Open64 and SmartHeap libraries used during compilation of the binaries. This environment variable setting is not needed when building the binaries on the system under test.

kernel/randomize_va_space

This option can be used to select the type of process address space randomization that is used in the system, for architectures that support this feature. 0 - Turn the process address space randomization off. This is the default for architectures that do not support this feature anyways, and kernels that are booted with the "norandmaps" parameter. 1 - Make the addresses of mmap base, stack and VDSO page randomized. This, among other things, implies that shared libraries will be loaded to random addresses. Also for PIE-linked binaries, the location of code start is randomized. This is the default if the CONFIG_COMPAT_BRK option is enabled. 2 - Additionally enable heap randomization. This is the default if CONFIG_COMPAT_BRK is disabled.

O64_OMP_SPIN_COUNT

Specify the number of times the spin loops will spin at user-level before falling back to operating system schedule/reschedule mechanisms. The default value is 20000.