SPEC CPU2006 Config Files

Last updated: $Date: 2014-04-09 11:19:43 -0400 (Wed, 09 Apr 2014) $ by $Author: JohnHenning $

(To check for possible updates to this document, please see http://www.spec.org/cpu2006/Docs/ )

Overview

Click one of the following to go to the detailed contents about that item:

I. Introduction

II. Config file options for runspec

III. Config file options for specmake

IV. Config file options for the shell

V. Config file options for readers

VI. Using Feedback Directed Optimization (FDO)

VII. The config file preprocessor

VIII. Output files - and how they relate to your config file

IX. About Alternate Sources

X. Troubleshooting

Contents

I. Introduction

A. What is a config file? (Background: benchmark philosophy.)

B. What does a config file affect?

1. runspec

2. specmake

3. The shell

4. Readers of the results

5. The config file preprocessor

C. Config file structure

1. Comments and whitespace

2. Header section

3. Named sections, section markers, section specifiers

a. Precedence for the benchmark specifier

Default benchmark

Named suite

Named benchmark

Order of differing sections does not matter

Order of the same section does matter

b. Precedence for the tuning specifier

c. Precedence for the extension specifier

Extension found in config file

Extension not found in config file

Extension override

d. Combining specifier types

e. Precedence among section types

4. MD5 section

5. Shell-like "here documents" and backslash continued lines

6. Included files

D. Variable substitution

1. By runspec

a. At startup: $[variable]

b. During a run: $variable and ${variable}

c. Example: cleaning files before a training run

d. Example: submit to multiple nodes, Tru64 Unix

e. Example: bind to processors, SPEC TurboBlaster9000

2. By the shell \$VARIABLE

a. Protecting shell variables

b. Example: submitting to multiple nodes, SGI Origin 2000

3. By specmake $(VARIABLE)

4. Limitations on variable substitution

5. Unsetting a variable with "%undef%"

II. Config file options for runspec

A. Precedence: config file vs. runspec command line

B. Options

action allow_extension_override backup_config basepeak bind build_in_build_dir check_md5 check_version command_add_redirect copies delay deletework difflines env_vars expand_notes expid ext fail fail_build fail_run feedback flagsurl http_proxy http_timeout ignore_errors ignore_sigint info_wrap_columns inherit_from iterations keeptmp line_width locking log_line_width log_timestamp mach mail_reports mailcompress mailmethod mailport mailserver mailto make make_no_clobber makeflags max_active_compares mean_anyway minimize_builddirs minimize_rundirs no_input_handler no_monitor nobuild notes_wrap_columns notes_wrap_indent output_format output_root parallel_setup parallel_setup_prefork parallel_setup_type parallel_test parallel_test_submit plain_train preenv rate rebuild reportable runlist section_specifier_fatal sendmail setprocgroup size src.alt strict_rundir_verify sysinfo_program table teeout train_with tune use_submit_for_speed verbose version_url

C. About sysinfo

1. The example sysinfo program

a. Location and updates

b. Turning the feature on or off

c. Output

d. Avoiding warnings about conflicting fields

e. Example: where does the example sysinfo program send its output?

2. Writing your own sysinfo program

III. Config file options for specmake

CC, CXX, FC
CLD, CXXLD, FLD
ONESTEP
OPTIMIZE, COPTIMIZE, CXXOPTIMIZE, FOPTIMIZE
PORTABILITY, CPORTABILITY, CXXPORTABILITY, FPORTABILITY...
RM_SOURCES
PASSn_CFLAGS, PASSn_CXXFLAGS, PASSn_FFLAGS

The syntax "+=" is available (but should be used with caution)

About Automatic Rebuilds

IV. Config file options for the shell

A. Options

bench_post_setup fdo_pre0 fdo_preN fdo_make_cleanN fdo_pre_makeN fdo_make_passN fdo_post_makeN fdo_runN fdo_postN post_setup submit

B. Using and Changing fdo Options

C. Using submit

1. Basic Usage

2. Useful features related to submit

3. Continuation of Submit Lines

Sidebar: About Quoting and Submit

4. Reporting of submit usage

V. Config file options for readers

A. Descriptive fields

company_name hw_avail hw_cpu_name hw_cpu_char hw_cpu_mhz hw_disk hw_fpu hw_memory hw_model hw_nchips hw_ncores hw_ncoresperchip hw_ncpuorder hw_nthreadspercore hw_ocache hw_other hw_pcache hw_scache hw_tcache hw_vendor license_num machine_name prepared_by sw_auto_parallel sw_avail sw_base_ptrsize sw_compiler sw_file sw_os sw_parallel_defeat sw_parallel_other sw_state sw_other sw_peak_ptrsize tester tester_name test_date test_sponsor VENDOR

B. Field scoping and numbered continuation lines

C. Additional notes for the reader

1 Notes sections

notes_comp_NNN notes_port_NNN notes_base_NNN notes_peak_NNN notes_submit_NNN notes_os_NNN notes_plat_NNN notes_part_NNN notes_NNN

2 Note numbering

3 Additional tags

4 Links in notes sections

5 Attachments

D. About Auto Parallel Reporting

VI. Using Feedback Directed Optimization (FDO)

A. The minimum requirement: PASSn_<language>FLAGS, PASSn_<language>OPTIMIZE, or fdo*n

B. Combining PASS*n and fdo*n; fake is your friend

C. Interaction with the config file feedback option

D. If the config file feedback option is used at multiple levels

E. Interaction with runspec --feedback

F. More examples

VII. The config file preprocessor

A. Defining macros

B. Un-doing macro definition

C. Using macros

D. Conditionals

1. %ifdef .. %endif

2. %ifndef .. %endif

3. %if .. %endif

4. %else

5. %elif

E. Informational directives

1. %warning

2. %error

VIII. Output files - and how they relate to your config file

A. Automatic backup of config files

B. The log file and verbosity levels

1. Useful Search Strings

2. About Temporary Debug Logs

3. Definitions of verbosity levels

C. Log file example: Feedback-directed optimization.

D. Help, I've got too many logs

E. Finding the build directory

F. Files in the build directory

G. For more information

IX. About Alternate Sources

A. Example: Applying a src.alt

B. Developing a src.alt (brief introduction)

X. Troubleshooting

 

Note: links to SPEC CPU2006 documents on this web page assume that you are reading the page from a directory that also contains the other SPEC CPU2006 documents. If by some chance you are reading this web page from a location where the links do not work, try accessing the referenced documents at one of the following locations:

I. Introduction

SPEC CPU2006 config files provide very detailed control of testing. Before learning about these details, most users will find it helpful to begin with:

The runspec document discusses the primary user interface for running SPEC CPU2006; with this document, attention turns more toward how things work inside.

q. This document looks big and intimidating. Where do I start?

a. Don't start here. Start with runspec.html. But, once you do read this document, please be sure to notice:

  • Section I.B, which explains that a config file contains options that are used by five (5) different consumers.
  • The table of contents, which lists the options accepted by each of those consumers. For example, the shell options are listed here.

If you keep track of which options are addressed to which consumer, you will considerably ease your learning curve.

I.A. What is a config file? (Background: benchmark philosophy.)

A config file contains:

A key decision that must be made by designers of a benchmark suite is whether to allow the benchmark source code to be changed when the suite is used.

If source code changes are allowed:

 +  The benchmark can be adapted to the system under test.
 +  Portability may be easier.
 –  But it may be hard to compare results between systems, unless some formal audit is done to ensure that comparable work is done.

If source code changes are not allowed:

 +  Results may be easier to compare.
 –  It may take more time and effort to develop the benchmark, because portability will have to be built in ahead of time.
 –  Portability may be hard to achieve, at least for real applications. Simple loops of 15 lines can port with little effort, and such benchmarks have their uses. But real applications are more complex.

SPEC has chosen not to allow source code changes for the CPU2006 suite, except under very limited circumstances.

By restricting source code changes, SPEC separates the activity of porting benchmarks, which has a goal of being performance neutral, from the activity of using the benchmarks, where the goal is getting the best score possible. Prior to the first production use of CPU2006, SPEC invested substantial effort to port the suite to as many platforms as practical, including 32 and 64-bit systems; little-endian and big-endian hardware; Unix, Linux, Mac OS X, and Microsoft Windows systems.

Are source code changes ever allowed? Normally, no. But if you discover a reason why you believe such a change is essential, SPEC wants to hear about it, and will consider such requests for a future revision of the suite. SPEC will normally not publish CPU2006 results using modified source code, unless such modifications are unavoidable for the target environment, are reviewed by SPEC, are made available to all users of the suite, and are formally approved by a vote.

So, if source code changes are not allowed, but the benchmarks must be compiled in a wide variety of environments, can the users at least write their own makefiles, and select -D options to select different environments? The answer to these two questions are "no", and "yes", respectively:

You do this in the config file, which contains a centralized collection of all the portability options and optimization options for all the benchmarks in the CPU2006 suite. The SPEC tools then automatically generate the makefiles for you.

The config file contains places where you can specify the characteristics of both your compile time and run time environments. It allows the advanced user to perform detailed manipulation of makefile options, but retains all the changes in one place so that they can be examined and reproduced.

The config file is one of the key ingredients in making results reproducible. For example, if a customer would like to run the CPU2006 suite on her own SuperHero Model 4 and discover how close results are in her environment to the environment used when the vendor published a CPU2006 result, she should be able to do that using only 3 ingredients:

  1. The SPEC CPU2006 media, ordered from SPEC.
  2. The software and hardware as specified in the full disclosure when the vendor published the original results.
  3. The config file from the vendor's submission.

I.B. What does a config file affect?

A config file contains options targetting five distinct consumers:

  1. The runspec tool, which provides overall control of the test,
  2. The tool that compiles benchmarks: specmake,
  3. Your shell, which is used for certain operations such as submiting to specific processors,
  4. Readers of the results, and
  5. The config file preprocessor.

To understand how to write a config file effectively, you need to understand which consumer you are addressing at any given point.

The above point seems worth emphasizing:

To understand how to write a config file effectively, you need to understand which consumer you are addressing at any given point.

This section gives you an overview of the consumers; more detail is included in later sections.

I.B.1. runspec

Various aspects of the operation of runspec can be affected by setting options within a config file. You'll find a list of these options in the table of contents for section II - including some that are available both on the runspec command line and some that can only be set within a config file.

For example, if michael.cfg includes the lines:

   output_format = asc,ps
   tune          = base
   reportable    = 1
   runlist       = fp

then the defaults for the runspec command would change as specified. A user who types either of the following two commands would get precisely the same effect:

   runspec --config=michael
   runspec --config=michael --output=asc,ps --tune=base --reportable fp 

I.B.2. specmake

The tool specmake is simply GNU make renamed to avoid any possible conflicts with other versions of make that may be on your system. The options commonly used for specmake are listed in the table of contents for section III.

For example, these config file lines:

   CC           = cc
   CPORTABILITY = -DSPEC_CPU_LP64
   OPTIMIZE     = -O4

are written to the makefile set that is ultimately used to build the benchmark, and are interpreted by specmake.

I.B.3. The shell

Some config file lines define commands that are handed off to the shell or, on Windows, the command interpreter. The list of these is in the table of contents for section IV.

For example, consider a config file that contains:

   fdo_pre0     = mkdir /tmp/joydeep/feedback; rm -f /tmp/joydeep/feedback/* 

When using this config file, runspec will pass the above command to the shell prior to running a training run for feedback directed optimization. It is the shell that actually carries out the requested commands, not runspec, so the syntax of the command depends on whether you are using the Unix shell (/bin/sh) or the Windows command interpreter (cmd.exe).

Because runspec can cause arbitrary commands to be executed, it is therefore important to read a config file you are given before using it.

I.B.4. Readers of the results

If a SPEC CPU2006 result is published (for example, at http://www.spec.org/), it is expected to contain all the information needed for a reader to understand exactly what was tested. Fields that are informative for readers are listed in the table of contents for section V.

For example, config file lines such as these are addressed to the human reader:

   hw_avail  = Apr-2008
   sw_avail  = May-2008
   notes_base_015  = Note: Feedback directed optimization was not used

In addition, for results published by SPEC, the config file itself is available to readers at http://www.spec.org/cpu2006/. The config file is presented as you wrote it, with three exceptions (protected comments, the MD5 section, and rawfile corrections for reader fields). The config file is made available because it is so important to reproducing results, as described in the Introduction. The config file is saved on every run, as a compressed portion of the rawfile, and can be accessed with runspec --rawformat --output_format=config <rawfile>.

I.B.5. The config file preprocessor

There is also a config file preprocessor, which is addressed via lines that begin with % in the first column. (History: The config file preprocessor was added in SPEC CPU2006 V1.0.)

I.C. Config file structure

A config file contains:

About "scope": Every line of the config file is considered to be within the scope of one of the above three. Lines prior to the first section marker are in the scope of the header section. All other lines are either in the scope of the most recently preceding user-defined section marker, or else in the MD5 section.

A line within the scope of a named section may be overridden by a line within the scope of a different named section, according to rules that are described below.

I.C.1. Comments and whitespace

Comment lines begin with a # (also known as a "hash mark"), and can be placed anywhere in a config file. You can also add comments at the end of a line, as shown below.

#
# This config file contains new tuning for the new C++ compiler
# 
iterations = 1   # For quick testing.  Reportable runs use 3.
OPTIMIZE = -O11  # Most go up to ten.  These go to eleven.

If you need to include a hash mark that you do NOT want to be treated as the start-of-comment delimiter, put a backslash in front of it:

hw_model = Mr. Lee's \#1 Semi-Autonomous Unit

When the config file is saved as an encoded portion of the rawfile, the comments are included. But if a comment line begins with #>, it is a "protected comment" that will not be saved in the rawfile. Thus you could use # for most of your comments, and use #> for proprietary information, such as:

   #> I didn't use the C++ beta version because of Bob's big back-end bug.

Blank lines can be placed anywhere in a config file.

Trailing spaces and tabs are stripped, unless they are preceeded by a backslash:

   CC_PATH=/path/with/no/trailing/spaces  

That is turned into "/path/with/no/trailing/spaces". To preserve those trailing spaces, you'd simply add a backslash:

   CC_PATH=/path/with/trailing/spaces\ 

That is turned into "/path/with/trailing/spaces ".

Spaces within a line are usually ignored. Of course, you wouldn't want to spell OPTIMIZE as OPT I MIZE, but you are perfectly welcome to do either of the following:

   OPTIMIZE=-O2
   OPTIMIZE   =  -02

One place where spaces are considered significant is in notes, where the tools assume you are trying to line up your comments in the full disclosure reports. (Notes are printed in a fixed-width font.)

Spaces at the beginning of lines are ignored, except when attempting to address the preprocessor. Preprocessor directives always begin with a percent sign (%) in the first column. You can put spaces after the percent sign, if you wish, as shown by the examples below.

I.C.2. Header section

The header section is simply the first section, prior to the first occurrence of a named section.

Most attempts to address runspec itself must be done in the header section. For example, if you want to set reportable=1, you must do so before any occurrences of section markers.

Relaxed Header Restrictions: New with CPU2006 V1.2, options that are restricted to appear only in the header section may nevertheless be entered in a named section, if and only if that section's name is default=default=default=default: (or a shorter spelling thereof, such as just default:).

It is, usually, still a good idea to put these options up top, since they tend to be global in nature and config files are typically more maintainable when global options are kept together.

Nevertheless, there may be circumstances where it is more convenient to, effectively, 'return' to the header section, for example, if multiple preprocessor macros want to write text into both the header section and named sections.

(Internally, the tools still think of your option as a header section option, even if you take advantage of this relaxed syntax and enter it into default=default=default=default: instead. Therefore this document, also retains the "header section" terminology.)

I.C.3. Named sections, section markers, section specifiers

A named section is a portion of the config file that begins with a section marker and continues until the next section marker or the MD5 section is reached. The contents of the named section are applied based upon the precedence rules described in the following sections.

A "section marker" is a one- to four-part string of the form:

   benchmark[,...]=tuning[,...]=extension[,...]=machine[,...]:

These are referred to below as the 4 "section specifiers". The allowed values for the section specifiers are:

benchmark: default
int
fp
Any individual benchmark, such as 465.tonto
A list of benchmarks, such as 450.soplex,403.gcc (History: The ability to specify more than one benchmark was added with SPEC CPU2006 V1.0.)
A benchmark set, using one of the bset names found in $SPEC/benchspec/CPU2006 or %SPEC%\benchspec\CPU2006 (History: The ability to specify bset names was added with SPEC CPU2006 V1.0, and is documented as of SPEC CPU2006 V1.1.)
tuning: default
base
peak
A list of tuning levels, such as base,peak (History: The ability to specify more than one tuning level was added with SPEC CPU2006 V1.0, and is documented as of SPEC CPU2006 V1.1.)
extension: default
An arbitrary string, such as "cloyce-naturalblonde"
A list of extensions, separated by commas (History: The ability to specify more than one extension was added with SPEC CPU2006 V1.0, and is documented as of SPEC CPU2006 V1.1.)
machine: default
An arbitrary string [*]
A list of machine types, separated by commas (History: The ability to specify more than one machine type was added with SPEC CPU2006 V1.0, and is documented as of SPEC CPU2006 V1.1.)

Trailing default section specifiers may be omitted from a section marker. Thus all three of these section markers are equivalent:

   465.tonto=base=default=default:
   465.tonto=base=default:
   465.tonto=base:

[*] The "machine" specifier works similarly to the extension specifier, but it does not affect the name of the executable produced, which, in many environments, makes it less useful than the extension specifier. This document does not describe details of the usage of the "machine" specifier, other than to note that it exists; if you feel particularly courageous, you may experiment with it.

Warning: Please be aware that if you use a single config file to build with two different machine settings, you will likely overwrite the original binaries each time you change the machine, since the machine specifier does not affect the name of the generated executable.

An example of the usage of the machine specifier may be found in your config directory, in Example-solaris-sparc-sunstudio.cfg. That config file applies -DSPEC_CPU_LP64 when the machine native64 is selected; it also takes steps to work around the limitation mentioned in the previous paragraph by using the config file preprocessor to change the extension.

Section markers can be entered in any order. Section markers can be repeated; material from identical section markers will automatically be consolidated. That is, you are welcome to start one section, start a different one, then go back and add more material to the first section.

By constructing section markers, you specify how you would like your options applied, with powerful defaulting and overriding capabilities. The next several sections walk through examples to demonstrate precedence, including how sections interact with each other.

I.C.3.a. Precedence for the benchmark specifier

For the benchmark specifier, the precedence is:

  highest   named benchmark(s)
            suite name
  lowest    default

Using default as the benchmark specifier

For example, consider this config file that only mentions default for the benchmark specifier:

   $ cat tmp.cfg
   runlist       = sjeng
   size          = test
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1

   default=default=default=default:
   OPTIMIZE = -xO1 -w

   $ runspec --config=tmp | grep sjeng.c
   cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG   -xO1 -w    sjeng.c
   $ 

The config file above is designed for quick, simple testing: it runs only one benchmark, namely 458.sjeng, using the smallest (test) workload, runs it only once, uses only base tuning, outputs only the text-format (ASCII) report, and displays the build commands to the screen (teeout). To use it, we issue a runspec command, and pipe the output to grep to search for the actual generated compile command. (Alternatively, on Windows, we could use findstr on the generated log file).

The careful reader may ask, "Why does the runlist reference sjeng rather than 458.sjeng?" The answer is that the runlist can use benchmark numbers or any abbreviation that is sufficient for uniqueness; any of the following would have the same effect: "458.sjeng", "458", "sjeng", "sj", "458.sj". This is the same rule as for the corresponding option on the command line.

The results show that the tuning applied was the expected -xO1 -w (which mean optimization level 1 and suppress warnings, for the Sun Studio Compilers). The tools have automatically added -c -o sjeng.o to specify where the object file is to be written. The switches -DSPEC_CPU -DNDEBUG were also added automatically. The former may enable benchmark code changes from SPEC's porting process (if any), and the latter turns off C language "assert" statements (if any).

Using a named suite as the benchmark specifier

The next example differs from the previous one by adding a section marker with int, for the integer suite, as the first section specifier:

   $  cat tmp.cfg
   runlist       = sjeng
   size          = test
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1

   default=default=default=default:
   OPTIMIZE = -xO2 -w

   int=default=default=default:
   OPTIMIZE = -xO3 -w

   $ runspec --config=tmp | grep sjeng.c
   cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG   -xO3 -w     sjeng.c
   $

The second OPTIMIZE line is used above because the reference to the integer suite is considered to be more specific than the overall default.

(History: documented since SPEC CPU2006 V1.1): You can also mention the "bsets" found in $SPEC/benchspec/CPU2006 or %SPEC%\benchspec\CPU2006. For example, you could reference all the C++ benchmarks by saying:

   all_cpp=default=default=default:
   OPTIMIZE = -xO3 -w

You can mention multiple sets:

   all_c,all_cpp=default=default=default:
   OPTIMIZE = -xO3 -w

Using a named benchmark as the benchmark specifier

Furthermore, we can add a specifier that mentions sjeng by name:

   $  cat tmp.cfg
   runlist       = sjeng
   size          = test
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1

   default=default=default=default:
   OPTIMIZE = -xO2 -w

   int=default=default=default:
   OPTIMIZE = -xO3 -w

   458.sjeng=default=default=default:
   OPTIMIZE = -xO4 -w

   $ runspec --config=tmp | grep sjeng.c
   cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG   -xO4 -w      sjeng.c
   $

The third OPTIMIZE line wins above, because it is included in the section that is considered to be the most specific.

(History: Documented as of SPEC CPU2006 V1.1): You can name more than one benchmark if you wish, as in:

   401.bzip2,458.sjeng=default=default=default:
   OPTIMIZE = -xO4 -w

Order of differing sections does not matter:

Let's change the example from the previous section to a different order.

   $ cat tmp.cfg
   runlist       = sjeng
   size          = test
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1

   458.sjeng=default=default=default:
   OPTIMIZE = -xO4 -w

   default=default=default=default:
   OPTIMIZE = -xO2 -w

   int=default=default=default:
   OPTIMIZE = -xO3 -w

   $ runspec --config=tmp | grep sjeng.c
   cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG   -xO4 -w      sjeng.c

Notice above that the order of entry is not significant; it's the order of precedence from least specific to most specific.

Order of the same section does matter:

When a specifier is listed more than once at the same descriptive level, the last instance of the specifier is used. Consider this case:

   458.sjeng=default=default=default:
   OPTIMIZE = -xO4

   400.perlbench=default:
   OPTIMIZE = -fast

   458.sjeng=default=default=default:
   OPTIMIZE = -xO3

The ending value of OPTIMIZE for 458.sjeng is -xO3, not -xO4.

I.C.3.b. Precedence for the tuning specifier

For the tuning specifier, either base or peak has higher precedence than default.

Here is an example of its use:

   $ cat tmp.cfg
   runlist       = sjeng
   size          = test
   iterations    = 1
   tune          = base,peak
   output_format = text
   teeout        = 1

   default=default=default=default:
   CC       = /opt/SUNWspro/bin/cc -w

   default=base=default=default:
   CC       = /update1/bin/cc -w

   default=peak=default=default:
   CC       = /update2/bin/cc -w

   $ runspec --config=tmp | grep sjeng.c
   /update1/bin/cc -w -c -o sjeng.o -DSPEC_CPU -DNDEBUG    sjeng.c
   /update2/bin/cc -w -c -o sjeng.o -DSPEC_CPU -DNDEBUG    sjeng.c
   $ 

In the above example, we compile sjeng twice: once for base tuning, and once for peak. Notice that in both cases the compilers defined by the more specific section markers have been used, namely /update1/bin/cc and /update2/bin/cc, rather than /opt/SUNWspro/bin/cc from default=default=default=default.

I.C.3.c. Precedence for the extension specifier

For the extension specifier, any named extension is at a higher precedence level than the default.

Using an extension found in the config file

The next example, builds with either the library for the "bsdmalloc" memory allocation routines, or the multi-threaded "mtmalloc" routines. (Note: the SPEC CPU2006 benchmarks do not require multi-threading, and do not contain threading directives; but a compiler is allowed to choose to try to automatically decompose threads, and there's nothing in the run rules to prohibit you from linking with your preferred version of a supported malloc library.)

   $ cat tmp.cfg
   runlist       = sjeng
   size          = test
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1

   default=default=default:
   LIBS = -lslowmalloc

   default=default=myke:
   LIBS = -lbsdmalloc

   default=default=yusuf:
   LIBS = -lthread -lmtmalloc
   $
   $ runspec --config=tmp --extension=myke | grep sjeng.o
   cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG                    sjeng.c
   cc  attacks.o book.o crazy.o draw.o ecache.o epd.o eval.o leval.o moves.o 
   neval.o partner.o proof.o rcfile.o search.o see.o seval.o sjeng.o ttable.o 
   utils.o     -lbsdmalloc          -o sjeng
   $
   $ runspec --config=tmp --extension=yusuf | grep sjeng.o
   cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG                    sjeng.c
   cc  attacks.o book.o crazy.o draw.o ecache.o epd.o eval.o leval.o moves.o 
   neval.o partner.o proof.o rcfile.o search.o see.o seval.o sjeng.o ttable.o 
   utils.o     -lthread -lmtmalloc          -o sjeng
   $ 
   $ cd $SPEC/benchspec/CPU2006/458.sjeng/exe
   $ ls -lt | head -3
   total 5888
   -rwxrwxr-x   1 alan  staff      244688 May 11 16:32 sjeng_base.yusuf
   -rwxrwxr-x   1 alan  staff      244628 May 11 16:31 sjeng_base.myke
   $ 

Notice above that two different versions of sjeng were built from the same config file, and neither one used slowmalloc, since the named extension is higher priority than the default. Both executables are present in the exe directory for 458.sjeng.

Using an extension that is not found in the config file

The previous section demonstrated use of the runspec switch --extension to select among extensions defined in the config file. But what if the extension on the command line is not mentioned in the config file? The example below continues immediately from the example just above:

   $ runspec --config=tmp --extension=yusoff
   runspec v4200 - Copyright 1999-2006 Standard Performance Evaluation Corporation
   ...
   ERROR: The extension 'yusoff' defines no settings in the config file!
          If this is okay and you'd like to use the extension to just change
          the extension applied to executables, please put
       allow_extension_override = yes
          into the header section of your config file.

By default, if you mention an extension on the runspec command line that does not exist in the config file, the tools refuse to build.

Extension override

But if you add allow_extension_override=yes to the top of the config file, then the tools will build or run with the extension you specified, using the same settings as they would have used if no extension had been entered on the runspec command line.

The next example continues with a config file that adds allow_extension_override to the previous example config file:

   $ diff tmp.cfg tmp2.cfg
   0a1
   > allow_extension_override=yes
   $
   $ runspec --config=tmp2 --extension=yusoff | grep sjeng.o
   cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG                    sjeng.c
   cc attacks.o book.o crazy.o draw.o ecache.o epd.o eval.o leval.o moves.o 
   neval.o partner.o proof.o rcfile.o search.o see.o seval.o sjeng.o ttable.o 
   utils.o           -lslowmalloc          -o sjeng
   $
   $ cd $SPEC/benchspec/CPU2006/458.sjeng/exe
   $ ls -lt | head -2
   total 6384
   -rwxrwxr-x   1   alan   staff        244628 May 11 16:39 sjeng_base.yusoff

Notice above that although the tools now consent to build with the requested extension, the library settings now falls back to the default, since "yusoff" does not match any section markers.

(History: for CPU2000, the tools silently accepted any extension, including typos, which sometimes led to surprising results. CPU2006 tries to reduce surprises by rejecting your typos, unless you set allow_extension_override.)

I.C.3.d. Combining specifier types

If more than one section applies to a particular benchmark without disagreement among them, then all are applied.

Consider this example:

   $ cat tmp.cfg
   runlist       = sjeng
   size          = test
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1

   default=default=default=default:
   OPTIMIZE = -xO2 -w
   CC       = /opt/SUNWspro/bin/cc 
   LIBS     = -lbsdmalloc

   458.sjeng=default=default=default:
   OPTIMIZE = -xO4 -w

   default=peak=default=default:
   CC       = /update1/bin/cc 

   default=default=mt=default:
   LIBS     = -lthread -lmtmalloc

   $ runspec --config=tmp --tune=peak --ext=mt | grep sjeng.o
   /update1/bin/cc  -c -o sjeng.o -DSPEC_CPU -DNDEBUG   -xO4 -w  sjeng.c
   /update1/bin/cc   -xO4 -w attacks.o book.o crazy.o draw.o ecache.o epd.o 
   eval.o leval.o moves.o neval.o partner.o proof.o rcfile.o search.o see.o 
   seval.o sjeng.o ttable.o utils.o    -lthread -lmtmalloc         -o sjeng
   $ 

Notice above that all three sections applied: the section specifier for 458.sjeng, the specifier for peak tuning, and the specifier for extension mt.

I.C.3.e. Precedence among section types

If sections conflict with each other, the order of precedence is:

     highest     benchmark
                 suite
                 tuning
     lowest      extension

And this order can be demonstrated as follows:

    $  cat tmp.cfg
     runlist       = sjeng
     size          = test
     iterations    = 1
     tune          = base
     output_format = text
     teeout        = 1
     makeflags     = -j30
  
     default=default=default=default:
     OPTIMIZE = -xO1 -w
  
     int=default=default=default:
     OPTIMIZE = -xO2 -w
  
     458.sjeng=default=default=default:
     OPTIMIZE = -xO3 -w
  
     default=peak=default=default:
     OPTIMIZE = -xO4 -w
  
     default=default=mt=default:
     OPTIMIZE = -xO5 -w
     $
     $ runspec --config=tmp sjeng | grep sjeng.c  
[1]  cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG   -xO3 -w        sjeng.c
     $ runspec --config=tmp --tune=peak sjeng | grep sjeng.c
[2]  cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG   -xO3 -w        sjeng.c
     $ runspec --config=tmp --extension=mt sjeng | grep sjeng.c
[3]  cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG   -xO3 -w        sjeng.c
     $ 
     $ runspec --config=tmp --tune=base bzip | grep bzip2.c
[4]  cc -c -o bzip2.o -DSPEC_CPU -DNDEBUG   -xO2 -w        bzip2.c
     $ runspec --config=tmp --tune=peak bzip | grep bzip2.c
[5]  cc -c -o bzip2.o -DSPEC_CPU -DNDEBUG   -xO2 -w        bzip2.c
     $ runspec --config=tmp --extension=mt bzip | grep bzip2.c
[6]  cc -c -o bzip2.o -DSPEC_CPU -DNDEBUG   -xO2 -w        bzip2.c
     $ 
     $ runspec --config=tmp --tune=base sphinx3 | grep utt.c
[7]  cc -c -o utt.o -DSPEC_CPU -DNDEBUG -I. -DSPEC_CPU -DHAVE_CONFIG_H  -I. -Ilibutil -xO1 -w   utt.c
     $ runspec --config=tmp --tune=peak sphinx3 | grep utt.c
[8]  cc -c -o utt.o -DSPEC_CPU -DNDEBUG -I. -DSPEC_CPU -DHAVE_CONFIG_H  -I. -Ilibutil -xO4 -w   utt.c
     $ runspec --config=tmp --extension=mt sphinx3 | grep utt.c
[9]  cc -c -o utt.o -DSPEC_CPU -DNDEBUG -I. -DSPEC_CPU -DHAVE_CONFIG_H  -I. -Ilibutil -xO5 -w   utt.c
     $ runspec --config=tmp --tune=peak --extension=mt sphinx3 | grep utt.c
[10] cc -c -o utt.o -DSPEC_CPU -DNDEBUG -I. -DSPEC_CPU -DHAVE_CONFIG_H  -I. -Ilibutil -xO4 -w   utt.c
     $ 

Notice above that the named benchmark always wins: lines [1], [2], and [3]. If there is no section specifier that names a benchmark, but there is a section specifier that names a suite, then the suite wins: lines [4], [5], and [6]. As a floating point benhcmark, sphinx3 base uses only the default tuning in line [7]. Since there are no applicable benchmark or suite specifiers, then tuning or extension can be applied in lines [8] and [9]. If both tuning and extension are applied, tuning wins [10].

I.C.4. MD5 Section

The final section of your config file is generated automatically by the tools when the benchmarks are compiled, and looks something like this:

   __MD5__
   458.sjeng=base=none=default:
   # Last updated Thu May 11 08:53:12 2006
   optmd5=a0104c0975ee3b341e73437843df57a9
   baggage=
   compile_options=\
   @eNpz9vcNsFJITlbQBaJ8hfyCksz8vGK9fAVdl+AAV+d454BQINPPxdUp1F1BQUG3wt9YQbdcAR3Y\
   FOeXFiWn2nE5Wyk4O9sqJScrgZn+Tl7+ASG2SiimK3EBrQQa7Obj6B4MlMO0SYnL30oBqNHT1zPK\
   FagCYqsSl4+nnzfYtVgcYpOflJWaXFJshySGsBPkGh8XiMOwmw1W4B8KciySSwGYpEmr
   exemd5=9b42a21f12b1fbb4e87cd9bacd781c43

The "MD5" is a checksum that ensures that the binaries referenced in the config file are in fact built using the options described therein. For example, if you edit the config file to change the optimization level for 458.sjeng, the next time the file is used for sjeng, the tools will notice the change and will recompile it.

You can optionally disable this behavior, but doing so is strongly discouraged. See the acerbic remarks in the description of check_md5, below.

If you would like to see what portions of your config file are used in computing the MD5 hash, runspec with --debug=30 or higher, and examine the log file.

For published results, the published config file (from rawformat --output_format=config) does not include the MD5 section.

I.C.5. Shell-like "here documents" and backslash continued lines

Shell-style "here documents" are supported for setting variables to multi-line values. Backslash-continued lines are also supported:

   $ cat tmp2.cfg
   expand_notes  = 1
   size          = test
   runlist       = sjeng
   iterations    = 1
   output_format = text

   foo =<<EOT

   This +
   is a +
   test +
   EOT

   bar = \
         and +\
         so  +\
         is  +\
         this+

   notes01 = $foo
   notes02 = $bar

   $ runspec --config=tmp2 | grep txt
           format: ASCII -> /ailuropoda/raj/spec/result/CINT2006.159.test.txt
   $ grep + ../result/*159*txt
       This +
       is a +
       test +
             and +
             so  +
             is  +
             this+
   $

Note: although the above forms of continued lines are supported, they are rarely used. The more common method of continuation is by appending a number to a field, as described in the section "Field scoping and numbered continuation lines".

I.C.6. Included files

It is possible to include another file in your config file. A typical use for this feature might be to keep all the software information in the main config file, but to include the hardware information about the current System Under Test (SUT) in another file. For example:

   $ cat tmp.cfg
   output_format = text
   iterations    = 1
   size          = test
   sw_compiler   = myC V1.0
   sw_avail      = Mar-2007
   runlist       = sjeng
   include: SUT.inc

   default=base:
   OPTIMIZE = -O
   $ cat SUT.inc
   hw_model = SuperHero IV
   hw_avail = Feb-2008

   $ runspec --config tmp | grep txt
           format: ASCII -> /manas/spec/result/CINT2006.160.test.txt
   $ grep avail ../result/*160.test.txt
    [...]       Hardware availability: Feb-2008
    [...]       Software availability: Mar-2007
   $ 

Notice above that the report mentions both the hardware and software dates.

I.D. Variable Substitution

You can do variable substitution using your config file. But, as described in the Introduction, the contents of a config file are directed to various consumers. Therefore, effective use of variable substitution requires you to be aware of which software is doing the substitution. Differing syntax is used for each.

%{macro_name}

The config file preprocessor does macro substitution. Macros to be substituted always use a percent sign and curly brackets: %{macro_name} See the preprocessor section for more information.

$[variable]

A dollar sign and square brackets cause runspec to do substitution immediately after reading the config file. This is useful for variables that don't change during a run, for example $[top] to refer to the directory where SPEC CPU2006 has been installed.

$variable
${variable}

A dollar sign and the name of a variable cause Perl variable interpolation to be used, for example $hw_avail. Curly brackets can be used to help the parser avoid confusion with contiguous other content: ${hw_avail}ish.

\$VARIABLE

If runspec generates an external command which is handed off to your shell (or command interpreter), the variable will be interpreted according to the rules of your shell. Such a variable might look like this: \$SPEC.

$(VARIABLE)

If the variable is handed off to specmake, it is interpreted by the usual rules of GNU make. Such variables might look like this: $(FOPTIMIZE).

q. Wait a minute... all these choices for substitution? Which one do I want?

a. You probably want the preprocessor.

Have a look at the example at the top of Section VII. If that looks like what you want, you're all set; otherwise, you'll have to think through which consumer you are addressing (runspec, specmake, or the shell) and pick your syntax accordingly.

I.D.1. By runspec

Substitution by runspec itself uses two different methods: the first is immediately after the config file is read, and the second is via Perl variable interpolation.

I.D.1.a At startup: $[variable]

Substitution for variables of the form $[variable] happens immediately after the config file is read. Any value that's set in the config file and is visible in the scope where the variable is used can be substituted. Because of the named section scoping restriction, if you want to use variable substitution to note your optimization flags, the notes for the individual benchmarks must be in those benchmarks' sections:

400.perlbench=peak=default=default:
PEAKFLAG=-gofast
# The following will turn into what you expect
notes_peak_400_1=I use $[PEAKFLAG]

400.perlbench=base=default=default:
BASEFLAG=-besafe
# The following will turn into what you expect:
notes_base_400_1=I use $[BASEFLAG]
# The following will NOT WORK:
notes_base_400_2=My brother likes $[PEAKFLAG]

You can't substitute variables that don't exist or whose values aren't known when the config file is read.

Wrong:

ext = foo

default:
OPTIMIZE = -xO2
notes01 = my ext is $[ext]

default=default=bar:
OPTIMIZE = -xO1
notes02 = my ext is $[ext]

This doesn't work because the sorting of which extensions to use doesn't happen until after the config file is processed. In this particular example, it's obvious (to you) what the value should be, but the tools aren't as clever as you are.

Perhaps the most useful variable is the one for the top of the SPEC CPU2006 tree, $[top], often found in contexts such as:

    flagsurl = $[top]/myflagsdir/myflagsfile.xml

Variables of possible interest might include:

configpath The location of your config file
dirprot protection that is applied to directories created by runspec
endian 4321 for big endian, 1234 for little
flag_url_base directory where flags files are looked up
OS unix or windows
os_exe_ext exe for windows, nil elsewhere
realuser the user name according to the OS
top the top directory of your installed SPEC CPU2006 tree
username the username for purposes of tagging run directories
uid the numeric user id

You can substitute for most options that you enter into the config file, including: action, allow_extension_override, backup_config, basepeak, check_md5, check_version, command_add_redirect, config, copies, delay, deletework, difflines, env_vars, expand_notes, expid, ext, fake, feedback, flagsurl, http_proxy, http_timeout, ignore_errors, ignore_sigint, info_wrap_columns, iterations, line_width, locking, log_line_width, mach, mail_reports, mailcompress, mailmethod, mailport, mailserver, mailto, make, make_no_clobber, makeflags, max_active_compares, mean_anyway, minimize_builddirs, minimize_rundirs, no_input_handler, no_monitor, notes_wrap_columns, notes_wrap_indent, output_format, output_root, plain_train, rate, rawformat, rebuild, reportable, runlist, section_specifier_fatal, sendmail, setprocgroup, size, strict_rundir_verify, sysinfo_program, table, teeout, tune, use_submit_for_speed, username, verbose, version_url.

You can also print out the value of additional variables that you may have created.

Here is a sample config file that illustrates square bracket variable substitution:

$ cat x.cfg
expand_notes = 1
runlist      = fp
action       = validate
myfriend     = jamiemeow
output_root  = /tmp
flagsurl     = $[top]/Docs/flags/flags-advanced.xml

notes01      = Today, I am running $[runlist] in $[top] on a $[OS] system 
notes02      = Today the flags file is $[flagsurl]
notes03      = Today, my favorite friend is $[myfriend]

$ runspec --config=x  --fakereportable | grep txt
        format: ASCII -> /tmp/result/CFP2006.002.txt
$ grep Today /tmp/result/CFP2006.002.txt
     Today, I am running fp in /spec/cpu2006 on a unix system 
     Today the flags file is /spec/cpu2006/Docs/flags/flags-advanced.xml
     Today, my favorite friend is jamiemeow

You can't use square brackets to substitute variables whose value changes during a run or build.

Wrong:

default=default=default=default:
#        I executed ''
notes0 = I executed '$[command]'

What did you expect?

Don't worry -- you'll receive a warning if you use a variable that the tools don't know about. It's up to you to heed them.

I.D.1.b During a run: $variable, ${variable}

The second round uses Perl variable interpolation. Only Perl scalars (denoted by a leading $) can be interpolated. For example, notes001 below uses the log file number (generated by the tools) and the hardware availability date (which was set directly):

$ cat tmp.cfg
runlist       = mcf
tune          = base
size          = test

iterations    = 1
output_format = asc

expand_notes  = 1
hw_avail      = May-2006
notes001      = This run is from log.$lognum with hw_avail $hw_avail

$ runspec -c tmp | grep txt
        format: ASCII -> /spec/david/result/CINT2006.029.test.txt
$ grep with ../result/*029*txt
     This run is from log.029 with hw_avail May-2006
$ 

In this case, $hw_avail could also have been substituted in the first round by writing it as $[hw_avail]. In general, for variable interpolation, the earlier the better.

To put text immediately after a variable, you need to make it possible for the parser to see the variable that you want, by using braces:

   % tail -2 tmp.cfg
   notes001 =You have done ${lognum}x runs tonight, go to bed.

   % runspec -c tmp | grep txt
      format: ASCII -> /john/result/CINT2006.103.test.txt
   % grep done /john/result/CINT2006.103.test.txt
      You have done 103x runs tonight, go to bed.

Interpolation won't always do what you wish it might do: for example, some variables are only defined at certain times, and your submit or notes line might be interpolated at a different time. When debugging a config files that uses variable interpolation, you will probably find --size test useful.

Some things that you might choose to interpolate include:

baseexe The first part of the executable name, which is <baseexe>_<tune>.<ext>. For example, in "libquantum_base.foo", baseexe is "libquantum".
benchmark The number and name of the benchmark currently being run, for example 462.libquantum
benchname The name of the benchmark currently being run, for example libquantum
benchnum The number of the benchmark currently being run, for example 462
benchtop The top directory for the benchmark currently being run, for example /spec/cpu2006/benchspec/CPU2006/462.libquantum
BIND A value from your bind list. This variable is actually interpreted by specinvoke, and cannot be spelled with braces. Say $BIND, do not say ${BIND}.
command The shell command line to run the current benchmark, for example ../run_peak_test_foo.0001/bzip2_peak.foo dryer.jpg 2 > dryer.jpg.out 2>> dryer.jpg.err
commandexe The executable for the current command, for example ../run_peak_test_foo.0001/bzip2_peak.foo
ext The extension for the benchmark being run
iter The current iteration number
logname The complete log file name, for example /spec/cpu2006/result/CPU2006.168.log
lognum The log file number, for example 168
SPECCOPYNUM The current copy number, when running a SPECrate run. This variable is actually interpreted by specinvoke, and cannot be spelled with braces. Say $SPECCOPYNUM, do not say ${SPECCOPYNUM}. The first copy is 0 (zero).
SPECUSERNUM Do not use. This is the older, obsolete, CPU2000 spelling for what is now called SPECCOPYNUM. If you use it, it will be silently ignored - no warning is printed.
tune The tuning for the benchmark being run (base or peak)
workload The current workload number (within the iteration)

For example, let's say you want to go for minimal performance. You might want to do this with the nice command. You can say:

   submit=nice 20 '$command'

and the $command gets expanded to whatever would normally be executed but with nice 20 stuck in front of it.

If you'd like a complete list of the variables that you can use in your commands (relative to the config file you're using), set runspec's verbosity to 35 or higher (-v 35) and do a run that causes a command substitution to happen, with expand_notes=1.

I.D.1.c Example: cleaning files before a training run

Perhaps a more useful example is this one, which directs the shell to clean files related to the current executable in the temporary directory, before a training run for feedback directed optimization:

   fdo_pre0     = mkdir /tmp/pb; rm -f /tmp/pb/${baseexe}*

NOTICE in this example that although the commands are carried out by the shell, the variable substitution is done by runspec.

I.D.1.d Example: submit to multiple nodes, Tru64 Unix

The following lines submit to multiple nodes on a Compaq AlphaServer SC Series Supercomputer running Tru64 Unix:

   submit= echo "$command" > dobmk; prun -n 1 sh dobmk
   command_add_redirect=1

In this example, the command that actually runs the benchmark is written to a small file, dobmk, which is then submitted to a remote node selected by prun. The parallel run command, prun, can execute multiple copies of a process, but in this case we have requested just one copy by saying -n 1. The SPEC tools will create as many copies as required.

The command_add_redirect is crucial. What happens without it?

$ head prun1.cfg
submit= echo "$command" > dobmk; prun -n 1 sh dobmk

ext            = prun
action         = validate
runlist        = libquantum
size           = test
use_submit_for_speed = 1
iterations     = 1        
ignore_errors  = 1
output_format  = text
$ runspec -c prun1.cfg 
...
$ cd $SPEC/benchspec/CPU2006/462.libquantum/run/run_base_test_prun.0000/
$ cat dobmk 
../run_base_test_prun.0000/libquantum_base.prun 33 5
$ 

Now let's use command_add_redirect, and see how dobmk changes:

$ diff prun1.cfg prun2.cfg 
1a2
> command_add_redirect=1
$
$ runspec -c prun1.cfg 
...
$ cd $SPEC/benchspec/CPU2006/462.libquantum/run/run_base_test_prun.0000/
$ cat dobmk 
../run_base_test_prun.0000/libquantum_base.prun 33 5 > test.out 2>> test.err
$ 

Notice that with command_add_redirect=1, the substitution for $command includes both the name of the executable and the file assignments for standard in, standard out, and standard error. This is needed because otherwise the files would not be connected to libquantum on the remote node. That is, the former generates [*]:

echo "libquantum_base.prun 33 5" > dobmk; 
prun -n 1 sh dobmk > test.out 2>> test.err

And the latter generates [*]:

echo "libquantum_base.prun 33 5 > test.out 2>> test.err" > dobmk; 
prun -n 1 sh dobmk

[*] The picky reader may wish to know that the examples were editted for readability: a line wrap was added, and the directory string .../run_base_test_prun.0000/ was omitted. The advanced reader may wonder how the above lines were discovered: for the former, we found out what runspec would generate by going to the run directory and typing specinvoke -n, where -n means dry run; in the latter, we typed specinvoke -nr, where -r means that $command already has device redirection. For more information on specinvoke, see utility.html.

I.D.1.e Example: bind to processors, SPEC TurboBlaster9000

The following lines bind copies to individual processors on a SPEC TurboBlaster 9000 running *nix:

   bind= 0, 4, 8, 12, 16, 20
   submit= procbind -p $BIND $command

In this example, the list of processors to bind is set using the bind facility, which is then used as an argument to the procbind command. With these settings, the first copy will be bound to processor 0, the second to processor 4, the third to processor 8, and so on. If more than six copies are run, the list is re-used starting from the beginning; that is, copy seven would also be bound to processor 0 and so on.

An equivalent way to say the same as the above would be

   bind0= procbind -p 0
   bind1= procbind -p 4
   bind2= procbind -p 8
   bind3= procbind -p 12
   bind4= procbind -p 16
   bind5= procbind -p 20
   submit= $BIND $command

The content of each of the bind settings need not be the same, or even similar.

Two important notes about the above submit commands:

  1. $BIND must be spelled exactly as it is in the examples above. It is a limitation of the tools that saying things like ${BIND} just won't work.
  2. For bind settings that contain whitespace or commas, the second form (multiple assignments) must be used. You can't say something like this:

    bind = "procbind -p 0", "procbind -p 4", "procbind -p 8"  <-- WRONG
    

I.D.2. By the shell \$VARIABLE

Substitution by the shell - or by the windows command interpreter - uses backslash dollar sign. The next two sections explain why both punctuation marks are required, and then provide an example.

I.D.2.a Protecting shell variables

Because Perl variables look a lot like shell variables, you need to specially protect shell variables if you want to prevent Perl from trying to interpret them. Notice what happens with the protected and unprotected versions:

$ cat tmp.cfg
runlist              = mcf
size                 = test
tune                 = base,peak
iterations           = 1

output_format        = asc
teeout               = 1
expand_notes         = 1
use_submit_for_speed = 1

default=peak=default=default:
submit = echo "home=$HOME; spec=$SPEC;" > /tmp/chan; $command

default=base=default=default:
submit = echo "home=\$HOME; spec=\$SPEC;" > /tmp/nui; $command
$ runspec --config=tmp > /dev/null
$ cat /tmp/chan
home=; spec=;
$ cat /tmp/nui
home=/home/chris; spec=/spec/cpu2006;
$ 

In the first submit command, $HOME and $SPEC were gobbled up by runspec. But since those are not the names of variables that can be interpolated, empty strings were the result. In the second command, the backslashes prevented runspec from interpreting the variables, so they were seen by the shell instead.

By default, submit is only applied to SPECrate runs. In this example, we used it for a SPECspeed run as well, by setting

use_submit_for_speed = 1 

in the configuration file.

I.D.2.b Example: submitting to multiple nodes, SGI Origin 2000

Here is a more complex example which uses substitution by both runspec and by the shell (the lines will be concatenated before execution):

   submit0= let "NODE=$SPECCOPYNUM/2"; 
   submit1= export NODE=/hw/nodenum/\$NODE; 
   submit2= let "CPU=2*(($SPECCOPYNUM+1)/2)-$SPECCOPYNUM "; 
   submit3= export CPU; 
   submit4= /usr/sbin/dplace -place \$SPEC/submit.pf -mustrun $command

(Note that the above example originated on an SGI Origin system where the shell uses let and dplace and other commands that might not apply to your shell.) Let's walk through how it works.

First, runspec substitutes the current copy number for $SPECCOPYNUM, and then passes the above command to the shell which does the substitutions for CPU and NODE. This example originated on an SGI Origin 2000 system, where there are two CPUs per node. Suppose that runspec is about to submit the copy for copy number 17. In that case, the above lines effectively cause::

   submit0= let "NODE=$SPECCOPYNUM/2";      # i.e. 17, so NODE=8
   submit1= export NODE=/hw/nodenum/\$NODE;      # Now NODE=/hw/nodenum/8
   submit2= let "CPU=2*(($SPECCOPYNUM+1)/2)-$SPECCOPYNUM ";  # and CPU=1
   submit3= export CPU; 
   submit4= /usr/sbin/dplace                     # So we execute dplace
            -place \$SPEC/submit.pf              # using $SPEC/submit.pf
            -mustrun $command                    # and the expected command

The desired $command is run under the control of submit.pf, with $NODE=/hw/nodenum/8 and $CPU=1. Here are the contents of the placement file, submit.pf:

   memories 1 in topology physical near $NODE
   threads 1
   run thread 0 on memory 0 using cpu $CPU

An important note about the above submit command:

I.D.3. By specmake: $(VARIABLE)

Variables with a dollar sign and parentheses, aka "round brackets", are substituted by specmake. For example:

    COMPILER_DIR=/usr/local/bin/
    CC=$(COMPILER_DIR)cc

For a more extensive example of variable substitution handled by specmake, see the SPEC CPU2000 example at www.spec.org/cpu2000/docs/example-advanced.cfg. Search that file for LIBS, and note the long comment which provides a walk-through of a complex substitution handled by specmake.

Deprecated feature alert: Although it is also possible to pass information to specmake using curly brackets: ${SPECMAKE}, this is not recommended. Instead, you should consistently use curly brackets to address runspec and round brackets to address specmake. It is possible that a future version of runspec may insist on interpolating curly brackets itself, rather than allowing specmake to do so.

I.D.4. Limitations on variable substitution

Once runspec hands control over to specmake or to the shell, the results of further substitution are invisible to runspec. For this reason, you can't say:

Wrong:

MYDIR         = /usr/gretchen/compilers
FC             = $(MYDIR)/f90
notes_comp_001 = compiler: $(FC)

However, there are a couple of ways to get around this restriction. The best way for global settings is to use the preprocessor:

%define MYDIR /usr/gretchen/compilers
FC             = %{MYDIR}/f90
notes_comp_001 = compiler: %{MYDIR}/f90

That leaves a little to be desired, though, doesn't it? If your Fortran compiler is changed to 'f2001', you still need to remember to change it in two places. You could of course define a whole macro for this:

%define MYFC /usr/gretchen/compilers/f90
FC             = %{MYFC}
notes_comp_001 = compiler: %{MYFC}

But what if you have a config file where FC might be set in multiple places, and you really want to know how it was set right here? In that case, use a combination of the preprocessor and variable substitution:

$ cat right_here.cfg 
%define MYDIR /usr/gretchen/compilers

default=base:
FC              = %{MYDIR}/f2001
notes_comp_001  = For base, we used this Fortran: $[FC] 

410.bwaves=peak:
FC             = %{MYDIR}/f77
notes_comp_002 = For 410.bwaves peak, we used this Fortran: $[FC]

$ runspec -c right_here.cfg --fakereportable fp | grep txt       
        format: ASCII -> /usr/gretchen/spec/result/CFP2006.179.txt
$ tail -15 ../result/*179.txt

                          Compiler Invocation Notes
                          -------------------------
     For base, we used this Fortran: /usr/gretchen/compilers/f2001 
     For 410.bwaves peak, we used this Fortran: /usr/gretchen/compilers/f77

    SPEC and SPECfp are registered trademarks of the Standard Performance
    Evaluation Corporation.  All other brand and product names appearing
    in this result are trademarks or registered trademarks of their
    respective holders.
-----------------------------------------------------------------------------
For questions about this result, please contact the tester.
For other inquiries, please contact webmaster@spec.org.
Copyright 2006 Standard Performance Evaluation Corporation
Generated on Wed Aug  2 09:39:53 2006 by SPEC CPU2006 ASCII formatter v4626
$ 

I.D.5. Unsetting a variable with "%undef%"

It is sometimes useful to be able to undo the setting of a variable that is defined in a lower-precedence section. This is easily accomplished using the special value '%undef%':

  $ cat gnana.cfg
   teeout = yes
   action = build
   runlist = bzip2

   default=default:
   OPTIMIZE = -O
   COPTIMIZE = -fast

   401.bzip2=peak:
   COPTIMIZE = %undef%

   $ runspec --config=gnana --tune=base | grep bzip2.c
   cc -c -o bzip2.o -DSPEC_CPU -DNDEBUG   -O -fast                bzip2.c
   $ runspec --config=gnana --tune=peak | grep bzip2.c
   cc -c -o bzip2.o -DSPEC_CPU -DNDEBUG   -O                 bzip2.c
   $ go bzip
   /spec/gnana/kit91/benchspec/CPU2006/401.bzip2
   $ cd run/build_peak_none.0000/
   $ grep OPT Make*
   Makefile.spec:COPTIMIZE        = 
   Makefile.spec:OPTIMIZE         = -O

As you can see in the peak compilation, the '-fast' flag was not present because the setting for COPTIMIZE had been deleted.

II. Config file options for runspec

This section documents options that control the operation of runspec itself.

II.A. Precedence: config file vs. runspec command line

In the list that follows, some items are linked to runspec.html, because they can be specified either in a config file, or on the runspec command line.

If options can be specified on both the runspec command line and in a config file, the command line will win vs. items specified in the header section, but will not win over items specified in named sections. Effectively, the order of precedence is:

   named sections (highest)
   command line
   header section (lowest)

For an example of these precedence rules in action, see section VI.E. on --feedback

II.B. Options

In the table that follows, the "Use In" column indicates where the option can be used:

H use only in header section
N use in a named section.
H,N can be used in both the header section and in named sections. The item can therefore be applied on a global basis, and/or can be applied to individual benchmarks.
Option Use In Default Meaning
action H validate What to do.
allow_extension_override H 0

The runspec command can use --extension to select among different options in a config file, as mentioned above. But what if the extension mentioned on the runspec command does not occur in any section marker? What should be done then?

  • If allow_extension_override is 0 (or no or false), the tools will print an error message and exit.
  • If allow_extension_override is 1 (or yes or true), then the tools will attempt to build (or run) the same binary they would have built by default, but will name the binary with specified extension. See the example above.

(History: the allow_extension_override feature was added with CPU2006.)

backup_config H 1 When updating the MD5 hashes in the config file, make a backup copy first. Highly recommended to defend against full-file-system errors, system crashes, or other unfortunate events.
basepeak H,N 0 Use base binary and/or base result for peak. If applied to the whole suite (in the header section), then only base is run, and its results are reported for both the base and peak metrics. If applied to a single benchmark, the same binary will be used for both base and peak runs, and the median of the base run will be reported for both.
Option Use In Default Meaning
bind H,N ''

List of values with which $BIND should be replaced in a submit command. This can be a simple comma (or white-space) separated list, such as:

bind = 0,1,2,3,4,5,6,7, 16,17,18,19,20,21,22,23     

If your line is too long, don't try to continue it by adding a numeral to 'bind'; that has a different meaning, as described a little below. Instead, you can continue it either by putting a backslash at the end of lines, or by using a here document. The above single line could equally well be expressed as:

bind = <<EOT
0,   1,   2,   3,   4,   5,   6,   7,
16, 17,  18,  19,  20,  21,  22,  23        
EOT

Note above that the trailing comma after the "7" is both optional and harmless. It can be present or not, as you wish; in either case, the sequence of values for this example will be 6, 7, 16, 17.

The bind values do not have to be a simple number. You can also use it to substitute longer strings that themselves contain whitespace or commas. To do so, use the format

bindN = string 

The example that follows effectively assigns 1 copy to processor 1, and 3 copies to processor 0.

bind0 = taskset -c 1 
bind1 = taskset -c 0 
bind2 = taskset -c 0 
bind3 = taskset -c 0 

submit    = $BIND $command

Like notes, the indices are not important and are used for ordering only.

If there are more copies than bind values, they will be re-used in a circular fashion. If there are more bind values specified than copies, then only as many as needed will be used. (History: The bind feature was added with SPEC CPU2006 V1.0. The ability to change bind on a per-benchmark basis was added in SPEC CPU2006 V1.1.)

Option Use In Default Meaning
build_in_build_dir H 1

When set, put build directories in a subdirectory named

benchspec/CPU2006/nnn.benchmark/build/build... (Unix) or
benchspec\CPU2006\nnn.benchmark\build\build... (Windows)

Specifying '0' will cause the build directories to be the same as in SPEC CPU2006 V1.0:

benchspec/CPU2006/nnn.benchmark/run/build... (Unix)
benchspec\CPU2006\nnn.benchmark\run\build... (Windows)

Why are build directories separated? Benchmarks are now built in directories named nnn.benchmark/build rather than under the benchmark's run subdirectory in order to make it easier to copy, backup, or delete build and run directories separately from each other. It may also make problem diagnosis easier in some situations, since your habit of removing all the run directories will no longer destroy essential evidence 10 minutes before the compiler developer says "Wait - what exactly happened at build time?".

If you prefer the SPEC CPU2006 V1.0 behavior, you can revert to it by setting build_in_build_dir to 0.

(History: The build_in_build_dir feature was added in SPEC CPU2006 V1.1.)

Option Use In Default Meaning
check_md5 H 1

Runspec uses MD5 hashes to verify that executables match the config file that invokes them, and if they do not, runspec forces a recompile. You can turn that feature off by setting check_md5=0.

Warning: It is strongly recommended that you keep this option at its default, '1' (that is, enabled). If you disable this feature, you effectively say that you are willing to run a benchmark even if you don't know what you did or how you did it -- that is, you lack information as to how it was built!

The feature can be turned off because it may be useful to do so sometimes when debugging (for an example, see env_vars, below), but it should not be routinely disabled.

Since SPEC requires that you disclose how you build benchmarks, reportable runs (using the command-line switch --reportable or config file setting reportable=1) will cause check_md5 to be automatically enabled.

check_version H 0 (1 for
reportable
runs)

When set, before doing a reportable run, runspec will download a small file (~15 bytes) from www.spec.org containing the current version of the suite and the date it was released, and check your copy vs. that file. In this way, you can be notified if the version of the suite that you're using is out-of-date.

Setting this variable to 0 will disable this check.

If you'd like to check a local file instead, you can modify version_url to point to your internal copy.

If you would like to check your version for a NON-reportable run, you will need to add --check_version to your command line. Setting check_version=1 in the config file only causes the check for reportable runs. (History: The check_version feature was added in SPEC CPU2006 V1.0.)

Option Use In Default Meaning
command_add_redirect H 0 If set, the generated $command will include redirection operators (stdout, stderr), which are passed along to the shell that executes the command. If this variable is not set, specinvoke does the redirection itself. This option is commonly used when using the submit command; please see the example in section I.D.1.d.
copies H,N 1

Number of copies. For base, the number of copies must be the same for all benchmarks, but for peak it is allowed to vary: for example, you could decide to run 64 copies of all benchmarks except 458.sjeng, which would run only 63.

(History: change introduced with SPEC CPU2006 V1.1) If you select basepeak=1 for a benchmark, the number of copies in peak will be forced to be the same as in base. In SPEC CPU2006 V1.0, selection of the number of copies could differ, but this was deemed to be an error. If you want to run the same tuning in both base and peak, while changing the number of copies, you will need to build two binaries with the same compiler switches.

delay H,N 0 Insert a delay of the specified number of seconds before and after benchmark execution. This delay does not contribute to the measured runtime of the benchmark. This delay is also not available in a reportable run. (History: The delay feature was added in SPEC CPU2006 V1.0.)
deletework H,N 0 If set to 1, always delete existing benchmark working directories. An extra-careful person might want to set this to ensure no unwanted leftovers from previous benchmark runs, but the tools are already trying to enforce that property.
difflines H,N 10 Number of lines of differences to print when comparing results.
Option Use In Default Meaning
env_vars H,N 0

If this variable is set to 1, then environment settings can be changed for benchmarks using ENV_* options in the config file. For example, consider the following executable, which fails because the library directory has been removed:

$ ls -l bwaves_base.tmp3
-rwxrwxr-x  1 david ptg 304964 May 16 9:16 bwaves_base.tmp
$ ldd bwaves_base.tmp | head -3
       libmvec.so.1 =>  /lib/libmvec.so.1
       libfui.so.2 =>   (file not found)
       libfai.so.3 =>   (file not found)
$ 

But if we add the following to the config file:

check_md5=0
env_vars=1
410.bwaves:
ENV_LD_LIBRARY_PATH=/spec/david/lovecraft/lib

then bwaves gets an LD_LIBRARY_PATH which points to the new library directory, and the benchmark runs successfully. (The addition of check_md5=0 allows us to change the settings for 410.bwaves without triggering a rebuild.)

Note that env_vars is not legal for a reportable run. The run rules require that the environment be set prior to the start of runspec. Nevertheless, the feature is available, since it might be useful during debugging, or for instrumentation purposes when doing performance analysis.

If you are trying to do a reportable run, perhaps the preenv feature is what you're looking for.

Which environment? If you are attempting to communicate settings from your shell environment into runspec, this is not the feature that you are looking for. Try the config file preprocessor instead. The env_vars option and ENV* are about communication from the config file to the environment of the invoked benchmark.

When developing a config file that uses env_vars, you may find it useful to set the verbosity level to 35 (or higher), which will cause the tools to log environment settings. (History: since SPEC CPU2006 V1.0, env_vars settings are logged.)

Option Use In Default Meaning
expand_notes H 0 If set, will expand variables in notes. This capability is limited because notes are NOT processed by specmake, so you cannot do repeated substitutions. You'll find some suggestions above.
expid H If set to a non-empty value, will cause executables, run directories, results, and log files to be put in a subdirectory (with the same name as the value set) in their normal directories. For example, setting expid = CDS will cause benchmark binaries to end up in exe/CDS, run directories to end up in run/CDS, and results and logs in $SPEC/result/CDS. (History: The expid feature was added with SPEC CPU2006 V1.0.)
ext H none Extension for executables created. This may not be set to any value that contains characters other than alphanumerics, underscores, hyphens, or periods.
Option Use In Default Meaning
fail H,N 0 If set, will cause a build or run to fail. (History: The ability to force a run to fail is new was added in SPEC CPU2006 V1.0.)
fail_build H,N 0

If set, will cause a build to fail. For example, you could say something like this:

400.perlbench=default:
#> I am posting this config file for use by others in the
#> company, but am forcing it to fail here because
#> I want to force users to review this section.
#>
#> Once you find your way here, you should test whether
#> bug report 234567 has been fixed, by using the first
#> line below.  If it has not been fixed, then use the
#> second.  In either case, you'll need to remove the
#> fail_build.
#>
#>   - Pney Guvaxre
#>     Boomtime, the 66th day of Confusion in the YOLD 3172

# OPTIMIZE = -Osuperduper
# OPTIMIZE = -Omiddling
fail_build = 1

In the example above, the build is forced to fail until the user examines and modifies that section of the config file. Notice that Pney has used protected comments to cause the comments about the internal bug report to disappear if the config file were to be published as part of a reportable run.

(History: The ability to force a build to fail was added in SPEC CPU2006 V1.0.)
fail_run H,N 0 If set, will cause a run to fail. (History: The ability to force a run to fail was added in SPEC CPU2006 V1.0.)
Option Use In Default Meaning
feedback H,N 1

The feedback option applies an on/off switch for the use of feedback directed optimization (FDO), without specifying how the feedback will be done.

  • If set to 0, then feedback will not be used, even if other options would normally cause it to occur.
  • If set to 1, then feedback is allowed to be used, provided that additional options are also set to describe how the FDO build should be carried out.

The interaction between feedback and these other options is described in section VI, below.

flagsurl H none

If set, retrieve the named URL or filename and use that as the "user" flags file. If the special value "noflags" is used, runspec will not use any file and (if formatting previously run results) will remove any stored file. (History: Automated prcoessing of flags was added in SPEC CPU2006 V1.0, and is explained in flag-description.html.)

If you want to list more than one flagsfile, the recommended method is by using numbered continuation lines, for example:

     flagsurl1 = mycompiler.xml
     flagsurl2 = myplatform.xml 

Using other methods (such as backslash continuation) to specify multiple flags files may appear to work, but may result in unexpected differences between the original config file and the config file as written by output format config.

New with CPU2006 V1.2, multiple flags files will typically be needed, because flags files are now separated into two types. See the overview in changes-in-v1.2.html.

Option Use In Default Meaning
http_proxy H

In some cases, such as when doing version checks and loading flag description files, runspec will use HTTP or FTP to fetch a file. If you need to specify the URL of a proxy server, this is the variable to use. By default, no proxy is used. Note that this setting will override the value of the http_proxy environment variable. For example, one might set:

http_proxy = http://webcache.tom.spokewrenchdad.com:8080 

Note: if an FTP proxy is needed, it must be set in the ftp_proxy environment variable; there is no corresponding config file setting.

Config files as posted at www.spec.org/cpu2006 will not include whatever you put on this line. (History: Support for http proxies was added in SPEC CPU2006 V1.0.)

http_timeout H 30 This is the amount of time (in seconds) to wait while attempting to fetch a file via HTTP or FTP. If the connection cannot be established in this amount of time, the attempt will be aborted. (History: Support for http timeout was added in SPEC CPU2006 V1.0.)
ignore_errors H 0 Ignore certain errors which would otherwise cause the run to stop. Very useful when debugging a new compiler and new set of options: with this option set, you'll find out about all the benchmarks that have problems, instead of only finding out about the first one.
ignore_sigint H 0 Ignore SIGINT. If this is set, runspec will attempt to continue running when you interrupt one of its child processes by pressing ^C (assuming that you have ^C mapped in the common way). Note that this does NOT cause runspec itself to ignore SIGINT.
Option Use In Default Meaning
info_wrap_columns H 50 When set to a value greater than 0, attempts to split non-notes informational lines such that they are no longer than info_wrap_columns columns wide. Lines are split on whitespace, and newly created lines are guaranteed to have at least the same indentation as the original line. If a line contains an item that is longer than info_wrap_columns, a warning is logged and the original line is left unchanged. (History: Automatic line wrapping was added in SPEC CPU2006 V1.0.).
inherit_from N ''

If set within a benchmark section, allows explicit inheritance of settings from another benchmark's section. The section to be inherited from is referenced using colons between the four section specifiers. Other inheritance mechanisms continue to work. Effectively, the referenced benchmark is the second highest priority -- second only to items specifically mentioned in the referring section. An example may help to clarify these points:

$ cat -n tmp.cfg
     1  iterations = 1
     2  size       = test
     3  teeout     = yes
     4
     5  default=default:
     6  OPTIMIZE = -xO1 
     7
     8  int=default:
     9  OPTIMIZE = -xO2
    10  LIBS     = -lbsdmalloc
    11
    12  401.bzip2=default:
    13  OPTIMIZE = -xO3 
    14  CC = /update1/cc
    15
    16  458.sjeng=peak:
    17  inherit_from = 401.bzip2:default:default:default
    18  CC = /update2/cc
$ runspec --config tmp --tune base sjeng | grep sjeng.o
cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG   -xO2        sjeng.c
cc  -xO2       attacks.o book.o ...  -lbsdmalloc  -o sjeng
$ runspec --config tmp --tune peak sjeng | grep sjeng.o
/update2/cc -c -o sjeng.o -DSPEC_CPU -DNDEBUG -xO3 sjeng.c
/update2/cc -xO3 attacks.o book.o ... -lbsdmalloc -o sjeng

In the above example,

  • The first runspec command uses the tuning for integer benchmarks, from the config file lines 8-10.
  • The second runspec command:
    • Still applies the integer tuning at line 10 (-lbsdmalloc).
    • Does not apply the optimization level on line 9, because it is overridden by other options. Line 17 says that we are to inherit from the bzip2 section, so the optimization level at line 13 (-xO3) is used instead.
    • But the compiler path at line 14 (/update1) is not used: line 18 (/update2) is highest priority.

Line 17 above could have been simplified. Just as trailing default section specifiers can be omitted at the original definition points, as explained above, they can also be omitted on an inherit_from option. (History: The inherit_from feature was added in SPEC CPU2006 V1.0.)

Option Use In Default Meaning
iterations H 3 Number of iterations to run.
keeptmp H 0

Whether or not to keep various temporary files. If you leave keeptmp at its default setting, temporary files will be automatically deleted after a successful run. If not, temporary files may accumulate at a prodigious rate, and you should be prepared to clean them by hand. Temporary files include:

  • CPU2006.nnn.log.debug in your result directory ($SPEC/result, %SPEC%\result, or the corresponding location under your output_root).

  • A directory named tmp under your top level SPEC directory, or under your output_root.

  • If tmp is not usable, you might find temporaries under tmp.cpu2006 or, possibly, tmp.cpu2006.n, where n may range from 1 to 15.

The keeptmp feature is new with SPEC CPU2006 V1.1.

line_width H 0 Line wrap width for screen. If left at the default, 0, then lines will not be wrapped and may be arbitrarily long.
locking H 1 Try to use file locking to avoid race conditions, e.g. if more than one copy of runspec is in use. Although performance tests are typically done with only one copy of runspec active, it can be handy to run multiple copies if you are just testing for correctness, or if you are compiling the benchmarks.
log_line_width H 0 Line wrap width for logfiles. If your editor complains about lines being too long when you look at logfiles, try setting this to some reasonable value, such as 80 or 132. If left at the default, 0, then lines will not be wrapped and may be arbitrarily long.
log_timestamp H 0 Whether or not to prepend time stamps to log file lines. (History: The log_timestamp feature was added in SPEC CPU2006 V1.1)
Option Use In Default Meaning
mach H default   Default machine ID. This may not be set to any value that contains characters other than alphanumerics, underscores, hyphens, or periods. See the warning in the description of section specifiers, above.
mailcompress H 0 When using the 'mail' output format, turning this on will cause the various report attachments to be compressed with gzip. (History: Report mailing was added in SPEC CPU2006 V1.0.)
mailmethod H smtp When using the 'mail' output format, this specifies the method that should be used to send the mail. On UNIX and UNIX-like systems, there are three choices: 'smtp' (communicate directly with an SMTP server over the network), 'mail' (try using mail(1) if available), and 'sendmail' (try invoking sendmail directly). On Windows systems, only 'smtp' is available. SMTP is the recommended setting. (History: Report mailing was added in SPEC CPU2006 V1.0.)
mailport H 25 When using the 'mail' output format, and when the mailmethod is 'smtp', this specifies the port to use on the mail server. The default is the standard SMTP port and should not be changed. (History: Report mailing was added in SPEC CPU2006 V1.0.)
mailserver H 127.0.0.1 When using the 'mail' output format, and when the mailmethod is 'smtp', this specifies the IP address or hostname of the mailserver through which to send the results. (History: Report mailing was added in SPEC CPU2006 V1.0.)
Option Use In Default Meaning
mailto H ''

The address or addresses to which results should be sent when using the 'mail' output format. If multiple addresses are specified, they should be separated by commas or whitespace. Each address should consist only of the name@domain part (i.e. no "full name" type info). The addresses are not checked for correct formatting; if a mistake is made, the results may be sent to an unknown location. Think: comp.arch. OK, probably not there, but seriously be careful about security on this one.

Config files as posted at www.spec.org/cpu2006 will not include whatever you put on this line (thus, spambots will not see the contents of this field).

Note that to get your reports mailed to you, you need to specify both mail as an output_format and an address to which they should be mailed. For example:

   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail

If no addresses are specified, no mail will be sent. (History: Report mailing was added in SPEC CPU2006 V1.0.)

mail_reports H all The list of report types to mail. The format and possible values are the same as for output_format, with the addition of log, which will cause the current log file to be sent. The default is for all files associated with the run to be mailed (so, this will include what you listed as your desired output_format plus log (the log file) and rsf (the rawfile). You can cut your email down to the bare essentials with something like this:
   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail
   mail_reports=text
If none of the requested report types were generated, no mail will be sent. (History: Report mailing was added in SPEC CPU2006 V1.0.)
Option Use In Default Meaning
make H,N specmake Name of make executable. Note that the tools will enforce use of specmake for reportable results.
make_no_clobber H,N 0 Don't delete directories when building executables. This option should only be used for troubleshooting a problematic compile. The tools will not allow you to use this option when building binaries for a reportable result. Note that you could issue multiple successive runspec commands with this option set (either in the config file, or with the --make_no_clobber switch), and the build directories will be preserved. But once you remove make_no_clobber (allowing it to default back to 0), then the tools will attempt a normal build with a fresh build directory.
makeflags H,N ''

Extra flags for make (such as -j). Set this to -j n where n is the number of concurrent processes to run during a build. Omitting n or setting it to zero unlimits the number of jobs that will be run in parallel. (Use of -j in conjunction with ONESTEP will still result in successful builds, but they will be necessarily serialized unless your compiler implements the parallelism itself.) Use with care! Other flags should be used here only if you are familiar with GNU make.

Note that requesting a parallel build with makeflags = -j N causes multiple processors to be used at build time. It has no effect on whether multiple processors are used at run time, and so does not affect how you report on parallelism.

(History: As of SPEC CPU2006 V1.1, you can use makeflags on Microsoft Windows)

Option Use In Default Meaning
max_active_compares   H,N # copies Max number of compares to run simultaneously while checking that the benchmarks computed acceptable answers. Useful when doing large SPECrate runs on a system with lots of CPUs.
mean_anyway H 0 Calculate mean even if invalid. DANGER: this will write a mean to all reports even if no valid mean can be computed (e.g. half the benchmarks failed). A mean from an invalid run is not "reportable" (that is, it cannot be represented in public as the SPEC metric).
minimize_rundirs H 0 Try to keep working disk size down. Cannot be used in a reportable run.
minimize_builddirs H 0 Try to keep working disk size down during builds.
nobuild H 0 Do not attempt to build benchmarks. Useful to prevent attempts to rebuild benchmarks that cannot be built. Also comes in handy when testing whether proposed config file options would potentially force an automatic rebuild. (History: The --nobuild feature was added in SPEC CPU2006 V1.0.)
no_monitor H,N '' Exclude the listed workloads from monitoring via the various monitor_* hooks. (History: The no_monitor feature was added in SPEC CPU2006 V1.0)
Option Use In Default Meaning
no_input_handler H,N close

Method to use to simulate an empty input. Choices are:

  • close - Just close stdin before executing the benchmark (this is the default).
  • zero - Create a zero-length file and attach it to stdin.
  • file - Synonym for zero.
  • null - Open the null device (/dev/null or NUL:) and attach it to stdin.

Normally, this option should be left at the default; it was actually added to the tools for the benefit of a different SPEC suite that needed the feature. If a reportable run for CPU2006 uses this feature, an explanation should be provided as to why it was used. (History: The no_input_handler feature was added in SPEC CPU2006 V1.0.)

notes_wrap_columns H 0 When set to a value greater than 0, attempts to split notes lines such that they are no longer than notes_wrap_columns columns wide. Lines are split on whitespace, and newly created lines are guaranteed to have at least the same indentation as the original line. If a line contains an item that is longer than notes_wrap_columns, a warning is logged and the original line is left unchanged. (History: Automatic line wrapping was added in SPEC CPU2006 V1.0.).
notes_wrap_indent H ' ' When line wrapping is enabled (see notes_wrap_columns), this is the string that will be prepended to newly created lines after the indentation from the original line is applied. The default is four spaces, but it can be set to any arbitrary string. (History: Automatic line wrapping was added in SPEC CPU2006 V1.0.)
Option Use In Default Meaning
output_format H all Format for reports. Valid options are listed at runspec.html under --output_format; major options include txt (ASCII text), html, pdf, and ps. You might prefer to set this to txt if you're going to be doing lots of runs, and only create the pretty reports at the end of the series. See also the information in runspec.html about --rawformat.
output_root H

When set to a non-empty value, causes all files written (other than config files) to be rooted in the directory named by the value, instead of being rooted in $SPEC. For example, setting output_root = /tmp/foo will cause results and logs to be deposited in /tmp/foo/result. This also applies to benchmark binaries and run directories.

This feature can be used to easily allow multiple people to access a single benchmark installation to which (with one exception) they do not need write access. The exception is $SPEC/config. Because the setting for the location comes from the config file, the config files still live under $SPEC. On Unix systems, you might choose to set the permissions on the config subdirectory to 1777 (which is the same as /tmp on many systems).

For an example of the use of output_root, see the section on it in runspec.html. (History: The output_root feature was added in SPEC CPU2006 V1.0.)

Option Use In Default Meaning
parallel_setup H 1

The number of run directories to setup simultaneously. For large SPECrate runs, directory setup takes significant time, as files for each copy are written and are checked for correctness. Using this feature may decrease the time required at the beginning of a large SPECrate run.

Parallelism is only applied per-benchmark: that is, all run directories will be setup for 400.perlbench before beginning to setup 401.bzip2, and so forth. Therefore, this option has no effect for SPECspeed runs or 1-copy SPECrate runs.

The maximum number of benchmark directory setups that will be done in parallel is equal to the number of copies being run.

A setting of "1" (the default) effectively disables this feature.

Notes:

  1. The parallel setup/test features (parallel_setup, parallel_setup_type, parallel_setup_prefork, parallel_test, and parallel_test_submit) control parallelism during the preparation phase for running the benchmarks, not the actual runs. Therefore, they have no effect on the setting of the report field "Auto Parallel: Yes/No", discussed below.
  2. These parallel setup/test features are not available on Microsoft Windows.
  3. Data may be written to stable storage using bench_post_setup and post_setup.
  4. A few statistics regarding potential benefit are included in changes-in-v1.1.html. Your performance will vary depending on how you have laid out your disk.
  5. When considering your disk layout options, bear in mind that the run rules require use of a single file system.
  6. (History: The parallel setup/test features were added in SPEC CPU2006 V1.1.)

The above notes apply to all the parallel_setup and parallel_test switches.

Option Use In Default Meaning
parallel_setup_prefork H none

If parallel_setup_type is set to fork, then the parallel_setup_prefork feature can be used to specify a command to be executed prior to doing the work in the forked child process. According to the run rules, the command must be one that causes a setup process to run on a particular processor.

The command must be able to bind a pre-existing process to a processor. The current process I.D. is available in the $$ variable. The $SPECCOPYNUM and $BIND variables are also available for substitution.

Example: if your operating system uses the "runon" command to assign work to a processor, you could enter the following to cause setup to use 8 threads, with processes assigned to the even numbered processors:

   bind                   = 0, 2, 4, 6, 8, 10, 12, 14 
   parallel_setup         = 8
   parallel_setup_type    = fork
   parallel_setup_prefork = runon $BIND $$ 

Note that we do not use any particular quoting methods in this case, because we do want runspec to interpret the variables $BIND and $$.

Technicality alert: the specified command is actually executed after the call to fork() that creates the new process, but before the forked child begins doing its work. Therefore, perhaps a more accurate name for the feature might have been something like parallel_setup_post_fork_pre_doing_the_work_within_the_fork, but such would exceed even the SPEC CPU2006 toolset's pedantry limit.

Notes: See notes under parallel_setup, above.

Option Use In Default Meaning
parallel_setup_type H fork

When doing parallel setup, select how the parallelism is accomplished. The possible values are:

  • none: Do no parallel setup, regardless of the setting for parallel_setup
  • fork: Use fork() to create setup processes. Use the command in parallel_setup_prefork (if any) to bind them to processors. This option has lower overhead than the next option, but is less flexible in terms of how jobs are distributed.
  • submit: Start a separate runspec invocation to do each setup, using the contents of the submit and bind variables as appropriate to distribute the jobs. Requires more overhead than the previous option, since runspec goes through its full initialization, including loading of perl modules.

Notes: See notes under parallel_setup, above.

parallel_test H #base
copies

For reportable runs, the tools verify that benchmark binaries get the correct answer for the test and train workloads. The time required for such verification does not count toward the reported score, but does add to the cost of benchmarking. To reduce that cost, you can run multiple tests simultaneously, by setting this option to anything higher than 1.

For SPECrate runs, the default is the number of base copies. For SPECspeed runs, the default is 1, that is, effectively off, but the feature can be turned on by setting it to something higher than 1.

If the feature is enabled, and if parallel_test_submit is also set, the settings for submit and bind will be used to distribute jobs if they're set.

Notes: See notes under parallel_setup, above.

parallel_test_submit H 0

Whether or not to use your submit and bind settings when doing parallel_test. When you specify parallel_test > 1, by default your submit and bind settings are not applied, because typically the operating system's default policies can be trusted to do an adequate job scheduling this (non-timed) work.

Notes: See notes under parallel_setup, above.

Option Use In Default Meaning
plain_train H,N 1 When true (set to 1, 'yes', or 'true'), does not apply any submit commands to the feedback training run. It also causes the monitor_* hooks to be ignored for the feedback training run. (History: The plain_train feature was added in SPEC CPU2006 V1.0.)
preenv H 1

Use preENV_ lines in the config file. When this option is set (the default), lines of the form preENV_<variable> = <value> will cause runspec to set the specified environment variable to value and re-exec runspec to perform the run. The restart is done in order to enforce the "unchanging environment" run rule.

Multiple preENV_ settings may appear in the config file.

For an example of the use of preENV settings, see the discussion of $LD_LIBRARY_PATH in the description of make_bundle.

Note that both the preenv option and any preENV_<variable> = <value> lines must appear in the header section of the config file, where the entire run is affected. If you are looking for a way to affect the environment of an individual benchmark, try env_vars.

(History: The preenv option was added in SPEC CPU2006 V1.1)

New with SPEC CPU2006 V1.2, the logging for preENV has changed to reduce confusion and to add useful information. The change is described in changes-in-v1.2.html.

Option Use In Default Meaning
reportable H 0 Strictly follow reporting rules, to the extent that it is practical to enforce them by automated means. The tester remains responsible for ensuring that the runs are rule compliant. You must set reportable to generate a valid run suitable for publication and/or submission to SPEC.
rate H 0 SPECrate vs SPECspeed testing. Specify 1 for SPECrate metrics, 0 for SPECspeed metrics.
rebuild H 0 Rebuild binaries even if they exist.
runlist H none What benchmarks to run. Names can be abbreviated, just as on the command line. See the long discussion of run order in runspec.html.
section_specifier_fatal H 1

While parsing the config file, if a section specifier is found that refers to an unknown benchmark or benchset, an error is output and the run stops. Set section_specifier_fatal=0 in the header section of your config file to convert this error into a warning and allow the run to continue. The ability to convert section specifier errors into warnings is probably of use only for benchmark developers.

(History: Prior to CPU2006, section specifier errors were silently ignored - which sometimes led to surprising results for testers. As of SPEC CPU2006 V1.0, by default, surprises are prevented by making the errors very visible.)

Option Use In Default Meaning
sendmail H /usr/sbin/
sendmail
When using the mail output format, and when the mailmethod is sendmail, this specifies the location of the sendmail binary. (History: The ability to mail reports was added in SPEC CPU2006 V1.0.)
setprocgroup H 1 Set the process group. On Unix-like systems, improves the chances that ^C gets the whole run, not just one of the children.
size H ref Size of input set. If you are in the early stages of testing a new compiler or new set of options, you might set this to test or train.
src.alt N ''

Name of subdirectory under <benchmark>/src/src.alt/ from which to draw approved source code modifications. Set this in the named section for the benchmark(s) where you wish to have src.alt(s) applied. Multiple src.alts may be specified for a single benchmark; the names should be separated by commas. If the specified alternate sources are not for the version of the suite in use, or if they are corrupt, the build will fail.

You may also spell this option as srcalt.
Any use of this feature is automatically noted in reports.
See also the chapter "About Alternate Sources", below.

(History: Automatic reporting of src.alts was added in SPEC CPU2006 V1.0.)

strict_rundir_verify H 1

When set, the tools will verify that the file contents in existing run directories match the expected MD5 checksums. Normally, this should always be on, and reportable runs will force it to be on. Turning it off might make the setup phase go a little faster while you are tuning the benchmarks. (History: SPEC CPU2000 provided fewer controls over the degree of verification.)

Developer notes: setting strict_rundir_verify=0 might be useful when prototyping a change to a workload or testing the effect of differing workloads. Note, though, that once you start changing your installed tree for such purposes it is easy to get lost; you might as well keep a pristine tree without modifications, and use a second tree that you convert_to_development.

Option Use In Default Meaning
sysinfo_program H 'specperl $[top]/Docs/sysinfo'

The name of an executable program or script that automatically fills out some of the information about your system configuration. An example sysinfo program is on your kit. To use the example, add this line near the top of your config file (i.e. in the header section):

sysinfo_program = specperl $[top]/Docs/sysinfo
(Note: this is the default for CPU2006 V1.2)

If you would like to turn the feature off, you can use:

sysinfo_program =

Warning: it is possible that SPEC may decide to require use of a sysinfo utility at a future time. If that occurs, the online copy of this document at www.spec.org/cpu2006/Docs/config.html will be updated to so indicate.

Details about the sysinfo utility may be found in the section About sysinfo, below, including how to selectively enable output types, how to resolve conflicting field warnings, and how to write your own sysinfo utility.

The example sysinfo program is substantially updated for CPU2006 V1.2

Option Use In Default Meaning
table H 1 In ASCII reports, include information about each execution of the benchmark.
teeout H 0 Run output through tee so you can see it on the screen. Primarily affects builds, but also provides some information about progress of runtime, by showing you the specinvoke commands.
train_with H,N train

Select the workload with which to train binaries built using feedback-directed optimization.

The ability to train with alternate workloads would not normally be applicable to CPU2006; the feature was added for the benefit of a different suite that uses the same toolset. Nevertheless, it could be used, for example when studying the efficacy of different training methods, as follows: (1) First convert your tree to a development tree; (2) place your new training workload under nnn.benchmark/data/myworkload. (3) Give it the same structure as the existing training workload: an input/ directory, an output/ directory, and a reftime file with contents similar to the one found in nnn.benchmark/data/train/reftime.

For reportable runs, you cannot use binaries that were trained with alternate workloads.

(History: The train_with feature was added in SPEC CPU2006 V1.1.)

Option Use In Default Meaning
tune H base default tuning level. In a reportable run, must be either all or base.
use_submit_for_speed H,N 0 If set, use submit commands for SPECspeed runs as well as SPECrate runs.
verbose H 5 Verbosity level. Select level 1 through 99 to control how much debugging info runspec prints out. For more information, see the section on the log file, below.
version_url H http://
www.
spec.org/
cpu2006/
current_
version
If version checking is enabled, this specifies the location from which the version information should be fetched.

II.C. About sysinfo

A sysinfo program is an executable program or script that automatically fills out some of the system description fields that are addressed to readers of your results. If you provide one, it will be called by runspec early during its execution.

Why sysinfo?

  1. The primary value of a sysinfo program is that it creates a contemporaneous record of the system under test.
  2. The record is created under the control of the SPEC toolset.
  3. Because system configuration information is automatically recorded, it is not subject to human transcription error.
  4. Therefore, a sysinfo program can remove ambiguity such as "I remember that I upgraded the version of the operating system, but I can't quite recall whether that was before or after I did the re-run of the SPECfp2006 test."

This section includes information about:

II.C.1. The example sysinfo program

New with CPU2006 V1.2, SPEC is providing a greatly expanded example sysinfo program, with support for several operating systems, including Linux, Mac OS X, Microsoft Windows, and Solaris.

Note that the example sysinfo program is enabled by default!

It is on by default because SPEC believes that the benefits listed in the box above, Why sysinfo?, outweigh its limitations.

SPEC is aware that there are many dependencies and differences among hardware and software platforms, and that these change from time to time. Thus its status as an "example": you may need to modify the example in order to use it on your system.

If your operating system is not recognized, the utility will print "Sorry, (name of os) is not supported by (name of program)."

II.C.1.a. Location and updates

The example sysinfo utility is on your kit, as $SPEC/Docs/sysinfo (Unix) or %SPEC%\Docs\sysinfo (Windows).

It is possible that SPEC may update the example from time to time; if so, updates will be posted at http://www.spec.org/cpu2006/Docs/sysinfo. (Note: the link goes to an executable file. If your web browser objects to opening it, try saving it instead.)

Version identification: The example sysinfo program, as supplied by SPEC, includes a revision and date of update. It generates its own md5sum when it is run. If you are unsure whether the copy in your Docs/ has been changed from SPEC's version:

  1. Make sure that $SPEC (Unix) or %SPEC% (Windows) is defined.
  2. Download the latest from www.spec.org/cpu2006/Docs/sysinfo.
  3. Run it.
  4. Compare the md5sum to the copy in your Docs/ directory.

For example (on a system where curl(1) is available to do the download):

$ . ./shrc
$ mkdir update
$ cd update
$ curl -s -O http://www.spec.org/cpu2006/Docs/sysinfo
$ chmod +x sysinfo
$ ./sysinfo -p | grep Rev | tr $ " "
#  Rev: 6743    Date:: 2011-08-11 #  6d78399eddfd6b1e8813c4ae4a352725
$ cd $SPEC/Docs
$ ./sysinfo -p | grep Rev | tr $ " "
#  Rev: 6743    Date:: 2011-08-11 #  6d78399eddfd6b1e8813c4ae4a352725
$ 

In the example above, the md5sum for the copy in the Docs directory matches the copy freshly retrieved from SPEC's server. (Note: when you try this for yourself, feel free to skip the tr(1) command at the end. It is there only to keep the dollar signs out of the above example text, which was deemed useful because otherwise SVN would automatically change the above example every time the document you are reading _now_ is checked in.)

II.C.1.b. Turning the feature on or off

To use the example sysinfo program, add this line near the top of your config file (in the header section):

sysinfo_program = specperl $[top]/Docs/sysinfo

(The above is the default for CPU2006 V1.2)

To turn it off, use:

sysinfo_program =

Warning about possible rules change: it is possible that, at a future time, SPEC may decide to require use of a sysinfo utility for reportable results. If that occurs, the online copy of this document at www.spec.org/cpu2006/Docs/config.html will be updated to so indicate.

Warning about config file re-use: When config files are re-used, for example, by download from www.spec.org, care must be taken to avoid accidental duplication of the sysinfo section and/or stale data. Therefore, prior to your run, please search your config file for the string sysinfo.

II.C.1.c. Output

By default, there will be more information in your platform notes. For example, by default Windows systems will include lines such as these:

                             Platform Notes
                             --------------
  Sysinfo program C:\kit118/Docs/sysinfo
  $Rev: 6894 $ $Date:: 2014-04-09 #$ \6d78399eddfd6b1e8813c4ae4a352725
  running on snowcrash Thu Aug 11 08:38:31 2011
 
  This section contains SUT (System Under Test) info as seen by
  some common utilities.  To remove or add to this section, see:
    http://www.spec.org/cpu2006/Docs/config.html#sysinfo
 
  Trying 'systeminfo'
  OS Name       : Microsoft Windows 7 Professional
  OS Version    : 6.1.7601 Service Pack 1 Build 7601
  System Manufacturer: LENOVO
  System Model  : 2537LB8
  Processor(s)  : 1 Processor(s) Installed.
       [01]: Intel64 Family 6 Model 37 Stepping 5 GenuineIntel ~2400 Mhz
  BIOS Version  : LENOVO 6IET75WW (1.35 ), 2/1/2011
  Total Physical Memory: 3,892 MB
 
  Trying 'wmic cpu get /value'
  DeviceID      : CPU0
  L2CacheSize   : 256
  L3CacheSize   : 3072
  MaxClockSpeed : 2400
  Name          : Intel(R) Core(TM) i5 CPU M 520 @ 2.40GHz
  NumberOfCores : 2
  NumberOfLogicalProcessors: 4
 
  (End of data from sysinfo program)

Also by default, the example sysinfo program will try to parse the above information into system description fields, such as:

hw_cpu_mhz         = 2400
hw_cpu_name        = Intel Core i5 M 520
hw_model           = 2537LB8
hw_nchips          = 1
sw_os000           = Microsoft Windows 7 Professional
sw_os001           = 6.1.7601 Service Pack 1 Build 7601

The output is written to a section named default: at the end of your config file. Therefore, by default, most of its output will be printed in all reports.

Editing the output: The example sysinfo program does not replace the need for human intelligence. Sysinfo creates contemporaneous evidence, and it may give you a start, but you will still have to understand your system, and you should examine its output carefully. If you see incomplete or incorrect information, please observe the following guidelines:

Original output is archived: The original output of the sysinfo utility may be found by running rawformat with output format config. If the description has been edited, rawformat will output both .cfg and .orig.cfg.

Quieter output: If you prefer, you can send the output to config file comments intead of reports, by adding one or both of these switches to the above sysinfo line:

   -f   Write comments instead of fields (e.g, 'sw_os' becomes '# sw_os')
   -p   Write comments instead of platform notes

Comment destination: Comments are written only to output format config. They do not appear in reports, and they are not written to the config file in your $SPEC/config directory. They are written to the copy in your $SPEC/result directory. This can make them easy to overlook, which is why the default is to write actual notes and fields, instead of writing only comments. See the example below for more information about output destinations.

II.C.1.d. Avoiding warnings about conflicting fields

When the example sysinfo utility is run, you may see warnings such as this one:

WARNING: Your config file sets some fields that are also set by sysinfo:
  hw_cpu_name, sw_os
To avoid this warning in the future, see
    http://www.spec.org/cpu2006/Docs/config.html#sysinfo

Any fields created by sysinfo (such as hw_cpu_name) are written to a section named default: at the end of your config file. The warning means that you have already set one of the listed fields in your default: section. Because the sysinfo output comes at the end of your config file, its setting will override the earlier config file settings, which may or may not be desirable.

To get rid of the warnings, you have four choices:

  1. You can remove the fields from your config file, and let the utility write them instead. To find out exactly which fields are written by the example sysinfo utility, search the source for the string:  promisedfields

  2. If you wish to allow the example sysinfo program to set some fields, while setting others yourself, remove the ones that sysinfo is allowed to write, and move the others to a section that has a higher precedence than the default: section. For example:

    int,fp=default=default=default:
    hw_memory001 = 8 GB (2 x 4 GB 2Rx8 PC3-10600E-9, ECC, 
    hw_memory002 = running at 1067 MHz and CL7)
    

    Notice in the above example that the first section marker is int,fp, which is higher priority than default. The effect will be to always silently apply the listed hw_memory001 and hw_memory002, since all benchmarks are either integer or floating point.

  3. If you wish to remove all the fields from the sysinfo output, you can use the -f switch, like so:

    sysinfo_program = specperl $[top]/Docs/sysinfo -f
  4. You can write your own sysinfo program, or you can modify the supplied example. Note: if you do so, you should retain a copy of the exact modified program together with your results.

II.C.1.e. Example: where does the example sysinfo program send its output?

An example may help to clarify which files get which output. On this system, $SPEC/config/dave.cfg runs the 401.bzip2 test workload once, with text and config outputs:

   $ cat dave.cfg
   runlist         = 401.bzip2
   size            = test
   iterations      = 1
   sysinfo_program = specperl $[top]/Docs/sysinfo 
   ext             = 0406
   output_format   = text,config

   default=default:
   CC              = /usr/bin/gcc 
   OPTIMIZE        = -O2 
   PORTABILITY     = -DSPEC_CPU_LP64
   $

A second config file differs only in that it adds the -p switch.

   $ diff dave.cfg radish.cfg
   4c4
   < sysinfo_program = specperl $[top]/Docs/sysinfo 
   ---
   > sysinfo_program = specperl $[top]/Docs/sysinfo -p
   $

Both config files are used, creating output file sets 20 and 21:

   $ for f in dave radish
   > do
   >     runspec -c $f | grep test.txt
   > done
       format: ASCII -> /home/dave/cpu2006/result/CINT2006.020.test.txt
       format: ASCII -> /home/dave/cpu2006/result/CINT2006.021.test.txt

On this Linux system, a line from /proc/meminfo about huge pages appears in several output files for run #20, but it is only a config file comment for run #21.

   $ go result
   /home/dave/cpu2006/result
   $ grep Hugep CINT2006.020.test*
   CINT2006.020.test.cfg:notes_plat_sysinfo_110 = Hugepagesize:     2048 kB
   CINT2006.020.test.rsf:spec.cpu2006.notes_plat_sysinfo_110:  Hugepagesize:     2048 kB
   CINT2006.020.test.txt:     Hugepagesize:     2048 kB
   $
   $ grep Hugep CINT2006.021.test*
   CINT2006.021.test.cfg:# Hugepagesize:     2048 kB
   $ 

II.C.2. Writing your own sysinfo program

Of course, you are not limited to using the pre-supplied sysinfo program from the Docs/ directory. You can write your own, by following these rules:

  1. The sysinfo program can be any executable program or script. It does not have to be a Perl script.
  2. If you do want to write a Perl script, you can count on specperl being on the path (as in the example above).
  3. The sysinfo program is run prior to most other config file processing. This means that most variables of the form $[var] and ${var} will not work
  4. You can use $[top] to find the top of your SPEC CPU2006 tree.
  5. The program should output lines of the form:
    name = value 
  6. The lines should be output to standard output.
  7. Only informational fields (hw_*, sw_*, notes*, test_sponsor, machine_name, license_num, tester, prepared_by) may be set; these settings will be inserted into the 'default=default=default=default' section, and will not be able to override any per-benchmark settings.
  8. Illegal lines will be silently discarded.
  9. Standard config file comments are also allowed.
  10. Because the line checking is fairly simple-minded, some config file features such as line continuation, block quotes, and line appending are not available.
  11. The output from the sysinfo program will be included in the final config as if it had been part of the original config file.
  12. Arguments can be given to the command if necessary.
  13. If the sysinfo_program fails to generate any output, the tools will write a warning to the log file. Errors and warning messages from your program might not be captured to the log.

III. Config file options for specmake

For a complete list of options that are used by specmake, please see makevars.html and notice which variables are documented as OK to mention in a config file.

Here are the commonly used variables:

CC How to invoke your C compiler.
CXX How to invoke your C++ compiler.
FC How to invoke your Fortran compiler.
(History: Prior to SPEC CPU2006 V1.0, there were options for Fortran 77. If you wish to use a Fortran 77 compiler for a benchmark in peak, set FC for that benchmark, not F77.)
   
CLD How to invoke the linker when compiling C programs.
CXXLD How to invoke the linker when compiling C++ programs.
FLD How to invoke the linker when compiling Fortran programs.
   
ONESTEP If set, build from sources directly to final binary. See the discussion in rule 2.2.10 of runrules.html.
   
OPTIMIZE Optimization flags to be applied for all compilers.
COPTIMIZE Optimization flags to be applied when using your C compiler.
CXXOPTIMIZE Optimization flags to be applied when using your C++ compiler.
FOPTIMIZE Optimization flags to be applied when using your Fortran compiler.
   
PORTABILITY Portability flags to be applied no matter what the compiler.
CPORTABILITY Portability flags to be applied when using your C compiler.
CXXPORTABILITY Portability flags to be applied when using your C++ compiler.
FPORTABILITY Portability flags to be applied when using your Fortran compiler.
FPPPORTABILITY Portability flags to be applied when pre-processing Fortran sources. Note that since preprocessing is not a part of standard Fortran, SPEC supplies a copy of the freely available filepp, with minor modifications, as specpp. Using specpp ensures that the preprocessing is done consistently across platforms. If you need to define Fortran preprocessor variables, do not put them in FPORTABILITY. Instead, put them in FPPPORTABILITY or PORTABILITY.
   
RM_SOURCES Remove a source file. Should only be used for library substitutions that comply with run rule 2.1.2.
   
PASSn_CFLAGS Flags for pass "n" C compilation when using feedback-directed optimization (FDO). Typically n is either 1 or 2, for the compile done before the training run and the compile done after the training run. See the chapter on Using Feedback Directed Optimization for more information.
PASSn_CXXFLAGS Flags for pass "n" C++ compilation when using FDO.
PASSn_FFLAGS Flags for pass "n" Fortran compilation when using FDO.
PASSn_LDFLAGS Flags to use with the linker in pass "n" when using FDO.

Note that you can also make up your own variable names, which specmake will use (and perform substitution on). For an example of this feature, see the SPEC CPU2000 example at www.spec.org/cpu2000/docs/example-advanced.cfg. Search that file for LIBS, and note the long comment which provides a walk-through of a complex substitution handled by specmake.

The syntax "+=" is available (but should be used with caution)

New with CPU2006 V1.2, the syntax "+=" is available to add to specmake options.

This option is available as a convenience to provide more flexible setting of options.

For example, consider this config file:

$ cat tmp.cfg
runlist = 401.bzip2

default:
OPTIMIZE       = -O1

int=default:
OPTIMIZE       += -unroll

default=peak:
OPTIMIZE       += -outer-unroll

401.bzip2=peak:
OPTIMIZE       += -inner-unroll

Notice how options accumulate based on the section specifiers above, and that the accumulation varies (as expected) depending on whether base or peak tuning is chosen:

$ runspec --config=tmp --tune=base --fake | grep bzip2.c   
cc -c -o bzip2.o -DSPEC_CPU -DNDEBUG   -O1 -unroll  bzip2.c
$ runspec --config=tmp --tune=peak --fake | grep bzip2.c
cc -c -o bzip2.o -DSPEC_CPU -DNDEBUG   -O1 -outer-unroll -unroll -inner-unroll  bzip2.c
$ 

Caution: although the option add to syntax flexibility when writing config files, that very flexibility may be a disadvantage. A config file that uses "+=" has the potential to introduce unwanted user surprises. Its effect will vary depending on precedence of section specifiers, as shown above; it will also vary depending on the order of your config file, and (if you mix and match features), there are glorious opportunities to surprise yourself by mixing += with macros, include files, and variable substititution. Don't do that. Keep it simple. Create conventions for your config files, write them down, in config file comments, and review the output (remember, --fake is your friend). Use the feature with caution, and drive responsibly.

About Automatic Rebuilds

(History: changed in SPEC CPU2006 V1.1)

Since the release of SPEC CPU2000, the SPEC CPU toolset has attempted to keep config files and binaries in synch with each other (unless you have set check_md5=0, which is not recommended). This generally means that edits to your config file have often resulted in the binaries being rebuilt, sometimes to the dismay of testers who found that rebuilds were attempted at inconvenient times. The inconvenience was compounded by the fact that the first step in a rebuild is to delete the old binary, so even a very fast interrupt of the run wouldn't help: once the rebuild would start, your binary was gone.

As of SPEC CPU2006 V1.1, unexpected rebuilds have been reduced.

In SPEC CPU2006 V1.0, the tools were much more likely to trigger a rebuild of the benchmark binaries than they are in later versions, because unrecognized options (e.g. a mis-spelled CXXOPTIMZIE, or a user-defined option such as MY_OPTS) would be passed to specmake, and the tools had no way to know what specmake did with such options. Since SPEC CPU2006 V1.1, the tools record only the options actually used by specmake, plus the options that are sent to the shell (e.g. via fdo_pre0), described in the next section). With this more careful recording, config file changes are less likely to trigger rebuilds, unless they actually affect the generated binary.

The tools do not promise to entirely eliminate unexpected rebuilds. Occasionally, they may make assumptions that seem conservative about config file options that might affect the generated binary. In particular:

Testing option sensitivity: If you would like to test whether a config file option change might cause a rebuild, without actually DOING the rebuild, here is a recommended procedure:

  1. copy the config file to a test version (let's call it: mytest.cfg)
  2. make the edits in the mytest.cfg
  3. make sure that mytest.cfg does NOT change the default for check_md5
  4. enter this command: runspec --config mytest --size test --iterations 1 --noignore_errors --nobuild ...

In the above command:

IV. Config file options for the shell

Some options in your config file cause commands to be executed by your shell (/bin/sh) or by the Windows command interpreter (cmd.exe).

Because runspec can cause arbitrary commands to be executed, it is therefore important to read a config file you are given before using it.

IV.A. Options

These are the options that cause commands to be executed:

bench_post_setup

Command to be executed after each benchmark's run directory setup phase. The run rules say that this feature may be used to cause data to be written to stable storage (e.g. sync). The command must be the same for all benchmarks. It will be run after each benchmark is setup, and for all workloads (test/train/ref), irrespective of the setting of parallel_test.

The bench_post_setup option may appear only in the header section.

(History: The bench_post_setup option was added in SPEC CPU2006 V1.1.)

fdo_pre0 Commands to be executed before starting a feedback directed compilation series.
fdo_preN Commands to be executed before pass N.
fdo_make_cleanN Commands to be executed for cleanup at pass N.
fdo_pre_makeN Commands to be done prior to Nth compile.
fdo_make_passN Commands to actually do the Nth compile.
fdo_post_makeN Commands to be done after the Nth compile.
fdo_runN Commands to be used for Nth training run.
fdo_postN Commands to be done at the end of pass N.
post_setup

Command to be executed after all benchmark run directories have been set up. The run rules say that this feature may be used to cause data to be written to stable storage (e.g. sync).

Notes:

  1. This option may only be specified in the header section.

  2. The post_setup option is not executed if minimize_rundirs is set.

  3. If you pick all for your runlist, post_setup is not executed separately for int and fp; it is executed only for the whole set of benchmarks. See the discussion of run lists in runspec.html.

  4. If parallel_test is greater than 1, post_setup is not executed for test and train.

Example 1: In a reportable run of all with parallel_test set to 1, you will get three instances of post_setup (see note 3 above).

Example 2: In a reportable run of all with parallel_test set to >1, you will get one instance of post_setup (see notes 3 and 4 above).

(History: The post_setup option was added in SPEC CPU2006 V1.1)

submit Commands to be used to distribute jobs across a multiprocessor system, as described below in the section on Using Submit

IV.B. Using and Changing fdo Options

Many examples of using the fdo_ options are provided in the chapter on Using Feedback Directed Optimization. A few introductory remarks are appropriate here:

Before setting an fdo option, you will want to observe the defaults, and of course your changes will have to comply with the run rules.

About fdo Defaults: To see the defaults for the various fdo commands, use the --fake switch. For example:

$ cat > test.cfg
OPTIMIZE = -O2
PASS1_CFLAGS = -feedback:collect
PASS2_CFLAGS = -feedback:apply
$
$ runspec --action build --fake -c test -T peak 462
...
specmake -n --always-make build FDO=PASS2 2> fdo_make_pass2.err | tee fdo_make_pass2.out

%% Fake commands from fdo_make_pass2 (specmake -n --always-make build FDO=PASS...):
cc -c -o classic.o -DSPEC_CPU -DNDEBUG    -feedback:apply    -O2                 classic.c
cc -c -o complex.o -DSPEC_CPU -DNDEBUG    -feedback:apply    -O2                 complex.c
...

In this example, we see that by default fdo_make_pass2 causes specmake to be invoked with --always-make build FDO=PASS2, which causes various cc compiler commands to be generated.

About changes to fdo options: You are allowed to modify the commands that are generated in order to support various compilation models. An example is provided in the chapter on Using Feedback-Directed Optimization. Of course, your modifications must comply with the run rules: for example, you could not use an fdo hook to get around the prohibition on mentioning names. If you are in doubt whether a change to an fdo hook is legal, please write to SPEC.

IV.C. Using submit

The config file feature submit allows you to enter commands that will distribute jobs across a multiprocessor system.

IV.C.1 Basic Usage

Normally submit is used with the config file variable $command plus your operating system's facilities that assign jobs to processors, such as dplace, pbind, procbind, prun, start/affinity, or taskset.

For example, this config file runs 2 copies of 401.bzip2 in SPECrate mode, assigning them to processors with the Linux taskset command:

$ cat tmp.cfg
copies   = 2
runlist  = 401.bzip2
rate     = 1
submit   = taskset -c $SPECCOPYNUM $command

default=base:
OPTIMIZE = -O2
$ runspec --config tmp 

In the example above, $SPECCOPYNUM acquires the values 0 and 1, causing processors 0 and 1 to be assigned the work of the 401.bzip2 commands.

IV.C.2 Useful features related to submit

Optionally, you may also wish to use the SPEC tool features bind, command_add_redirect, and/or $SPECCOPYNUM. To see how submit fits in with these tools features, please see the    several    detailed    examples    in the section on Variable Substitution, above.

You might also want to search published results at www.spec.org/cpu2006 for systems that are similar to your system. If you see SPECrate runs for such systems, the config files associated with them may have examples that combine the features in ways that are useful on your architecture.

IV.C.3 Continuation of Submit Lines

Continuation of multiple submit lines is supported, using continuation rules similar to other fields.

Here is an example from a Solaris system:

$ cat tmp.cfg
rate        = 1               # In SPECrate mode,
copies      = 2               # run 2 copies of
runlist     = 410.bwaves      # 410.bwaves, using 
size        = test            # the small "test" worklad.
iterations  = 1               # Just run it once.

command_add_redirect = 1      # Include redir. ops in $command

submit0 = echo 'pbind -b $SPECCOPYNUM \$\$ >> pbind.out' > dobmk
submit2 = echo "$command" >> dobmk
submit4 = sh dobmk

default=base:
OPTIMIZE = -O
$ runspec -c tmp2

As noted in the comments, the above example runs 2 copies of 410.bwaves with the "test" workload. On Solaris, processors are assigned work using the pbind command. The line with submit0 causes a small file, dobmk, to be created with the result of:

 echo 'pbind -b $SPECCOPYNUM \$\$ >> pbind.out'

That is, assign the current process to the processor whose number matches the current SPECCOPYNUM, and send the output of the pbind command to file pbind.out. The line with submit2 adds to dobmk the actual command that runs the benchmark, including any needed IO redirection (because command_add_redirect is set). Finally, the line with submit4 actually runs the benchmark.

(History: The feature to allow continuation of submit lines was added in SPEC CPU2006 V1.1.)

Sidebar: About Quoting and Submit

The above example uses all three common Unix types of quoting: backslashes, single quotes, and double quotes.

submit0 = echo 'pbind -b $SPECCOPYNUM \$\$ >> pbind.out' > dobmk
submit2 = echo "$command" >> dobmk
submit4 = sh dobmk

The details of the quoting may be more than you want to know, but if you develop or maintain submit options, you may want to carefully consider these types of quoting.

(1) The designator for the current process is quoted using backslashes \$\$ to prevent runspec from trying to interpret it. We do want runspec to interpret $SPECCOPYNUM, and it does so. Let's look at the lines that specinvoke actually runs: [excerpted, white space adjusted, line wraps added]:

$ go bwaves run
/export/home/kit107/benchspec/CPU2006/410.bwaves/run
$ cd run_base_test*0
$ specinvoke -nr
# Starting run for copy #0
echo 'pbind -b 0 $$ >> pbind.out' > dobmk;
echo "../run_base_test_none.0000/bwaves_base.none 2>> bwaves.err" >> dobmk;
sh dobmk

# Starting run for copy #1
echo 'pbind -b 1 $$ >> pbind.out' > dobmk;
echo "../run_base_test_none.0000/bwaves_base.none 2>> bwaves.err" >> dobmk;
sh dobmk
$ 

Notice above that $SPECCOPYNUM has become 0 and 1, but $$ remains unchanged by the SPEC tools.

(2) The second type of quoting is the set of single quotes that surround the phrase that is written to dobmk

echo 'pbind -b 0 $$ >> pbind.out' > dobmk

With single quotes, interpretation of $$ is delayed until dobmk itself is running: dobmk gets bound to a processor, and so does its child a moment later:

../run_base_test_none.0000/bwaves_base.none 2>> bwaves.err

Sidebar to the sidebar: If we use double quotes instead of single quotes here, an interesting bug happens. In that case, $$ gets interpreted by the parent shell and the parent pid is inserted. A little later, dobmk runs, and obediently binds its parent to a processor. It does not bind itself, though, and that causes a problem. ("Wait, wait", you say, "Bindings are inherited, so binding the parent should be fine, right?" Well, no. Bindings are inherited only for newly created processes. Binding your parent doesn't do a thing for yourself. So, you definitely want the $$ to be dobmk's PID, not its parent's.)

(3) Lastly, double quotes surround $command in the line

submit2 = echo "$command" >> dobmk

For this case, single or double would have been fine. The $command variable gets interpreted by the SPEC tools, and after that, there's no further interpretation needed.

Here are the contents of the dobmk files by the time all is done:

$ go bwaves run
/export/home/kit107/benchspec/CPU2006/410.bwaves/run
$ cat run_base_test*0/dobmk
pbind -b 0 $$ >> pbind.out
../run_base_test_none.0000/bwaves_base.none 2>> bwaves.err
$ cat run_base_test*1/dobmk
pbind -b 1 $$ >> pbind.out
../run_base_test_none.0000/bwaves_base.none 2>> bwaves.err
$ 

Hints: When debugging submit options, increase the log verbosity level. Capture standard output from your binding command -- pbind.out in this example.

It would also be a very good idea to capture standard error. The submit0 line above could be improved:

$ diff tmp.cfg better.cfg
9c9
< submit0 = echo 'pbind -b $SPECCOPYNUM \$\$ >> pbind.out' > dobmk
---
> submit0 = echo 'pbind -b $SPECCOPYNUM \$\$ >> pbind.out 2>> pbind.err' > dobmk
$

Use the command that allows you to query results of binding - e.g. pbind -q on Solaris [excerpted, white space adjusted]:

$ pbind -q
process id 23025: 0
process id 23029: 1
process id 23026: 1
process id 23030: 0
$ ps -ef | grep -e dobmk -e bwav
john 23025  0:00 sh dobmk
john 23029  0:12 ../run_base_test_none.0000/bwaves_base.none
john 23026  0:00 sh dobmk
john 23030  0:12 ../run_base_test_none.0000/bwaves_base.none

IV.C.4 Reporting of submit usage

If you use the submit feature, a notes section will automatically be created to indicate that you have done so.

                                 Submit Notes
                                 ------------
    The config file option 'submit' was used.

You can add notes to that section, or customize it as you wish, by creating lines with notes_submit_NNN. The phrase

The config file option 'submit' was used

must appear somewhere in your notes; if it doesn't, it will get added.

For example, if your config file has:

notes_submit_000 =
notes_submit_005 = Processes were assigned to specific processors using 'pbind' commands.  The config
notes_submit_010 = file option 'submit' was used, along with a list of processors in the 'BIND'
notes_submit_015 = variable, to generate the pbind commands.  (For details, please see the config file.)
notes_submit_020 =

then your report will include a section that looks like this:

                                 Submit Notes
                                 ------------
    
     Processes were assigned to specific processors using 'pbind' commands.  The config
     file option 'submit' was used, along with a list of processors in the 'BIND'
     variable, to generate the pbind commands.  (For details, please see the config file.)

Notice in the example that the key phrase (The config file option 'submit' was used) was wrapped across two lines. That's fine. You can break it in the middle, and you can vary the capitalization, just so long as you include it. If you don't include it, then it will be automatically added at the top of the Submit Notes section.

Note: now you see it, now you don't: The best part of this feature is that the entire section, including all your added notes, will automatically disappear if you do a run that does not use submit -- for example, a SPECspeed run (*). The intent is to make life easier for common usage of config files: when you do SPECrate runs, the section appears; when you do SPECspeed runs, it disappears (*), and you don't have to remember to edit the notes section each time that you switch between SPECspeed metrics and SPECrate metrics.

(*) (Assuming that you have kept the default setting of use_submit_for_speed = 0)

(History: The Submit Notes section was added in SPEC CPU2006 V1.1.)

V. Config file options for readers

Whether or not you send your result to SPEC, you should fully disclose how you achieved the result. If it requires the installation of the GoFastLinker, you should say so. By setting the appropriate fields in the config file, you can cause information about the GoFastLinker to appear in the reports that are intended for humans.

V.A. Descriptive fields

Here are the fields that you can set to describe your testbed to readers:

company_name Do not use. This field is obsolete. You are probably looking for hw_vendor, tester, or test_sponsor. See the example under test_sponsor.
hw_avail Date hardware first shipped. If more than one date applies, use the LATEST one.
hw_cpu_name Manufacturer-determined formal processor name.
hw_cpu_char Technical characteristics to help identify the processor. (You'll find more information about the intended use of hw_cpu_name vs. hw_cpu_char in the run rules.)
hw_cpu_mhz Speed of the CPUs, in MHz.
hw_disk

Disk subsystem for the SPEC run directories. Three important Notes:

  1. Only a single file system may be used for the run directories, as described in the run rules at Rule 3.1.1.
  2. If the output_root feature is used, its disk subsystem should be the one described here.
  3. Disks used for other purposes - such as the system disk - are usually not a performance-relevant variable, so need not be described. Note that if you wish to do so, you can extend this field using the same continuation rules as other fields (so, you could use hw_disk01, hw_disk02, etc).
hw_fpu Floating point unit.
hw_memory Size of main memory (and other performance-relevant information about memory, as discussed in the run rules.)
hw_model Model name.
hw_nchips Number of CPU chips configured. See the discussion of CPU counting in the run rules.
hw_ncores Number of CPU cores configured. See the discussion of CPU counting in the run rules.
hw_ncoresperchip Number of CPU cores per chip. See the discussion of CPU counting in the run rules.
hw_ncpuorder Valid number of processors orderable for this model, including a unit. For example, "2, 4, 6, or 8 chips".
hw_nthreadspercore Number of hardware threads per core. See the discussion of CPU counting in the run rules.
hw_other Any other performance-relevant hardware.
hw_pcache 1st level (primary) cache.
hw_scache 2nd level cache.
hw_tcache 3rd level cache.
hw_ocache 4th level or other form of cache.
hw_vendor The hardware vendor. An example of usage of this and related fields is given in the test_sponsor section.
license_num The SPEC license number for either the tester or the test_sponsor.
machine_name Do not use. This field is obsolete (and has been since SPEC CPU95 was retired).
prepared_by Is never output. If you wish, you could set this to your own name, so that the rawfile will be tagged with your name but not the formal reports.
sw_auto_parallel

(History: As of SPEC CPU2006 V1.1, the sw_auto_parallel field has been retired.) If your compiler creates binaries that use multiple processors, you should instead make sure that your flags file indicates this by setting the new flag attribute parallel="yes". The tools will notice if you use a flag that has the new flag attribute set, and if so will automatically include that information in the reports.

See the discussion of parallel reporting, below. See also the description for sw_parallel_other and sw_parallel_defeat.

sw_avail Availability date for the software used. If more than one date, use the LATEST one.
sw_base_ptrsize Size of pointers in base. Report:
  • "32-bit" if all benchmarks in base are compiled with switches that request only 32-bit pointers.
  • "64-bit" if all benchmarks in base are compiled with switches that request only 64-bit pointers.
  • "32/64-bit" if there is a mix of 32-bit and 64-bit. For example, if the C++ benchmarks use 32-bit pointers, while the Fortran and C benchmarks use 64-bit, then you would report "32/64-bit" here.
sw_compiler Name and version of compiler. Note that if more than one compiler is used, you can employ continuation lines. (This applies to most of the fields discussed here, but is emphasized for sw_compiler because it is common that testers first find themselves wanting to use continuation lines when documenting their compiler set.)
sw_file

File system (ntfs, ufs, nfs, etc) for the SPEC run directories. Three important Notes:

  1. Only a single file system may be used for the run directories, as described in the run rules at Rule 3.1.1.
  2. If the output_root feature is used, its file system should be the one described here.
  3. File systems used for other purposes - such as for the operating system - are usually not a performance-relevant variable, so need not be described. But if you wish to do so, you can extend this field using the same continuation rules as other fields (so, you could use sw_file01, sw_file02, etc).
sw_os Operating system name and version.
sw_other Any other performance-relevant non-compiler software used, including third-party libraries, accelerators, etc.
sw_parallel_defeat This field is only set in raw files, and can only be applied by rawformat. See the description in utility.html, and the discussion of parallel reporting, below.
sw_parallel_other

Yes or no. If this field is set to yes, then the run is assumed to have used auto-parallelization technology separate from the compilation system. Set this field to "yes" if the tested system uses libraries that spread work across multiple processors, or other non-compiler(*) means that cause individual benchmark executables to spread their work across multiple processors.

(This field can be specified in config files or in rawfiles, just like almost all other fields.)

* If your compiler does the parallelization, specify it in the flags file, using the parallel flag attribute.

See the run rules section on reporting of auto parallel, and the discussion of parallel reporting, below. (History: The sw_parallel_other feature was added in SPEC CPU2006 V1.1)

sw_peak_ptrsize Size of pointers in peak. Report:
  • "32-bit" if all benchmarks in peak are compiled with switches that request only 32-bit pointers.
  • "64-bit" if all benchmarks in peak are compiled with switches that request only 64-bit pointers.
  • "32/64-bit" if there is a mix of 32-bit and 64-bit.
sw_state Multi-user, single-user, default, etc. (History: effective as of the release of SPEC CPU2006 V1.1), the rules have been clarified regarding documentation of system state and tuning, in rules 3.1.2, 4.2.3 (paragraphs b and g), and 4.2.4 (paragraphs f, g).
tester The entity actually carrying out the tests. An optional field; if not specified, defaults to test_sponsor. An example is given in the test_sponsor section.
tester_name Do not use. This field is obsolete. You are probably looking for hw_vendor, tester, or test_sponsor. See the example under test_sponsor.
test_date When the tests were run. (History: since the release of SPEC CPU2006 V1.1), this field is populated automatically based on the clock in the system under test. Setting this in the config file will generate a warning and the setting will be ignored. If your system clock is incorrect, then the value may be edited in the raw file (see utility.html). It's better to avoid the necessity to edit, by setting your system clock properly.
test_sponsor

The entity sponsoring this test. An optional field; if not specified, defaults to hw_vendor.

For example, suppose that the Genius Compiler Company wants to show off their new compiler on the TurboBlaster 9000 computer, but does not happen to own a maxed-out system with eight thousand processors. Meanwhile, the Pawtuckaway State College Engineering department has just taken delivery of such a system. In this case, the compiler company could contract with the college to test their compiler on the big machine. The fields could be set as:

      test_sponsor = Genius Compilers
      tester       = Pawtuckaway State College
      hw_vendor    = TurboBlaster
VENDOR Do not use. This field is obsolete, and has been since SPEC CPU92 was retired. It is ancient, useless, vestigial, degenerate, deprecated, and leftover. Repetition is a form of emphasis.

V.B. Field scoping and numbered continuation lines

Fields can appear and disappear based upon scope. For example, you can define and document your C and C++ compilers for all runs, while mentioning Fortran only for floating point runs.

Most (*) fields can be continued to another line by appending a numeral, for example sw_os1, sw_os2, sw_os3 if you need 3 lines to fully identify your operating system.

Here is an example that uses both of these features:

   default=default=default=default:
   sw_compiler1 = TurboBlaster C/C++ V1.0
   CC           = cc -v      

   fp=default=default=default:
   sw_compiler2 = TurboBlaster Fortran V1.0
   sw_compiler3 = TurboBlaster F++ V1.0
   FC           = f90 -v

In the above example, the information about the C and C++ compilers will be printed for both integer and floating point runs. The information about the Fortran compilers will be printed only for floating point runs.

Note that the numerals above will be converted to a more formal style of three-digit numbers with leading zeros in the raw file; as discussed in utility.html, if you edit a rawfile you must use the more formal style of 000, 001, etc.

(*) The fields which cannot be continued are the ones that are expecting a simple integer, such as: hw_nchip, hw_ncores, hw_ncoresperchip, hw_nthreadspercore, license_num; and the ones that are expecting a date: hw_avail, and sw_avail.

V.C. Additional notes for the reader

In addition to the pre-defined fields, you can write as many notes as you wish. These notes are printed in the report, using a fixed-width font. For example, you can use notes to describe software or hardware information with more detail beyond the predefined fields:

   notes_os_001  = The operating system used service pack 2 plus patches 
   notes_os_002  = 31415, 92653, and 58979.  At installation time, the 
   notes_os_003  = optional "Numa Performance Package" was selected.

V.C.1 Notes sections

There are 8 different notes sections. (History: prior to SPEC CPU2006 V1.0, there was only one section.) If there are no notes in a particular section, it is not output, so you don't need to worry about making sure you have something in each section.

The sections, in order of appearance, are as follows:

  1. notes_comp_NNN -- Notes about compiler invocation.
  2. notes_port_NNN -- Notes about portability options.
  3. notes_base_NNN -- Notes about base optimization options.
  4. notes_peak_NNN -- Notes about peak optimization options.
  5. notes_submit_NNN -- Notes about use of the submit option.
  6. notes_os_NNN -- Notes about operating system tuning and changes.
  7. notes_plat_NNN -- Notes about platform tuning and changes.
  8. notes_part_NNN -- Notes about component parts (for kit-built systems).
  9. notes_NNN -- General notes.

Notes about the submit command are described above, with the description of the submit option. (History: the notes_submit feature was added with SPEC CPU2006 V1.1.)

V.C.2 Note numbering

Start your notes with the name of the notes section where you want the note to appear, and then add numbers to define the order of the lines. Within a section, notes are sorted by line number. The NNN above is not intended to indicate that you are restricted to 3 digits; you can use a smaller or larger number of digits as you wish, and you can skip around as you like: for example, ex-BASIC programmers might naturally use line numbers 100, 110, 120... But note that if you say notes_plat782348320742972403 you just might encounter the dreaded (and highly unusual) "out of memory" error, so don't do that.

You can optionally include an underscore just before the number, but beware: if you say both notes_plat_105 and notes_plat105, both are considered to be the same line. The last one mentioned will replace the first, and it will be the only one output.

V.C.3 Additional tags

For all sections you can add an optional additional tag of your choosing before the numbers. Notes will be organized within the tags. (Prior to CPU2006 V1.2, this feature was not available for the General notes.)

The intent of the feature is that it may allow you to organize your system information in a manner that better suits your own categories for describing it.

For example:

   $ cat tmp.cfg
   size                 = test
   iterations           = 1
   output_format        = text
   teeout               = 1
   runlist              = sjeng
   tune                 = base

   notes_part_greeting_011  = ++ how
   notes_part_greeting_20   = ++ you?
   notes_part_greeting_012  = ++ are
   notes_part_aname_1       = ++ Alex,
   notes_part_080           = ++ hi

   $ runspec --config tmp > /nev/dull
   $ cd ../result
   $ ls -t *test.txt | head -1
   CINT2006.101.test.txt
   $ grep ++ *101*txt
        ++ hi
        ++ Alex,
        ++ how
        ++ are
        ++ you?
   $ 

V.C.4 Links in notes sections

You can mention URLs in your notes section. (History: since the release of SPEC CPU2006 V1.1,) html reports will correctly render them as hyperlinks. For example:

notes_plat_001 = Additional detail may be found at
notes_plat_002 = http://www.turboblaster.com/servers/big/green/

If you like, you can use descriptive text for the link by preceding it by the word LINK and adding the descriptive text in square brackets:

LINK url AS [descriptive text]

The brackets may be omitted if your descriptive text is a single word, without blanks.

For example:

notes_plat_001 = Additional detail may be found at 
notes_plat_002 = LINK http://www.turboblaster.com/servers/big/green/ AS [TurboBlaster Servers]

When the above appears in an html report, it is rendered as:

Platform Notes

 Additional detail may be found at 
 TurboBlaster Servers

And in a text report, it appears as:

                                Platform Notes
                                --------------
     Additional detail may found at 
     TurboBlaster Servers (http://www.turboblaster.com/servers/big/green/)

Since the text report is not a context in which the reader can click on a link, it is spelled out instead. Note that because the text report spells the link out, the text line is wider than in HTML, PS, and PDF reports. When deciding where to break your notes lines, you'll have to pick whether to plan line widths for text (which may result in thin-looking lines elsewhere) or plan your line widths for HTML/PS/PDF (which may result in lines that fall of the right edge with text). The feature notes_wrap_columns won't help you here, since it is applied before the link is spelled out.

(History: the LINK AS feature was added in SPEC CPU2006 V1.1)

V.C.5 Attachments

You can cause files to be attached to a result with this syntax:

ATTACH url AS [descriptive text]

Unlike links (described in the previous section), if you use ATTACH, the mentioned file is actually copied into your result directory.

For example, the notes below will cause /Users/john/Desktop/power.jpg and /Users/john/Desktop/fan.jpg to be copied into the result directory:

notes_plat_110 = Stated performance depends on proper cooling, as shown in the 
notes_plat_120 = ATTACH file:///Users/john/Desktop/power.jpg AS [power supply photo] 
notes_plat_130 = and ATTACH file:///Users/john/Desktop/fan.jpg AS [fan diagram] 

When the above notes are used with an HTML report, they appear as:

Platform Notes

 Stated performance depends on proper cooling, as shown in the
 power supply photo 
 and fan diagram

And in a text report, they appear as:

                                Platform Notes
                                --------------
     Stated performance depends on proper cooling, as shown in the
     power supply photo (CINT2006.011.test.jpg) 
     and fan diagram (CINT2006.011.test.1.jpg)

In the text report, you can see that when fan.jpg and power.jpg were copied into the result directory, they were given names to show which result they correspond to: CINT2006.011.test.jpg and CINT2006.011.test.1.jpg. Note that since the text report spells the name of the attachment out, the text line is wider than in HTML, PS, and PDF reports. When deciding where to break your notes lines, you'll have to decide whether to plan line widths for text (which may result in thin-looking lines elsewhere) or plan your line widths for HTML/PS/PDF (which may result in lines that fall off the right edge with text). The feature notes_wrap_columns won't help you here, since it is applied before the attachment name is spelled out.

(History: the ATTACH feature was added in SPEC CPU2006 V1.1.)

V.D. About Auto Parallel Reporting

If benchmarks are automatically optimized to use multiple threads, cores, and/or chips at run time, the tester needs to indicate this in the report as

Auto Parallel: Yes

For SPEC CPU2006 V1.0, it was sometimes difficult to ensure that reports were accurate, in part because there can be multiple sources of run-time parallelism (as described in the run rules).

With SPEC CPU2006 V1.1, the V1.0 method of filling out the above field, sw_auto_parallel, has been retired, and three new features have been introduced to better reflect these sources of parallelism.

The intent is that the most common case will be handled automatically:

Overall, the setting of the Auto Parallel field in reports can be thought of as if it were derived from this logic:

   (i | ii) & (¬ iii)

... or for the benefit of those readers who think in FORTRAN, it is as if the derivation were:

  ( I .OR. II ) .AND. ( .NOT. III) 

VI. Using Feedback Directed Optimization (FDO)

Feedback Directed Optimization (FDO) is an application build method that involves:

The modified program is expected to run faster than if FDO had not been used. FDO is also sometimes known as PBO, for Profile-Based Optimization.

This section explains how various controls interact when using feedback. Those controls are:

  1. The specmake options PASS*n.
  2. The shell options fdo*n.
  3. The config file feedback option.
  4. The runspec switch --feedback.

VI.A. The minimum requirement: PASSn_<language>FLAGS, PASSn_<language>OPTIMIZE, or fdo*n

The default is to build without feedback. To use FDO, you must add either the PASSn_<language>FLAGS, PASSn_<language>OPTIMIZE, or fdo*n options. The PASSn options let you specify additional flags for the pre-defined sequence of steps within each build pass, where "n" indicates the build pass number. The fdo*n options let you add entire commands to build passes, including setup and cleanup commands.

VI.B. Combining PASS*n and fdo*n; fake is your friend

The PASSn* and fdo*n options can be freely used together. For example, here's a config file that builds 459.GemsFDTD peak (as specified on the first three lines), with both PASSn and fdo options:

$ cat tmp.cfg
action       = build
runlist      = GemsFDTD
tune         = peak

PASS1_FFLAGS = --CollectFeedback
PASS2_FFLAGS = --ApplyFeedback
fdo_run2     = $command
fdo_post2    = /usr/bin/postoptimizer

$ runspec --config tmp --fake | \
   grep -e excite.f90 -e train.err -e postopt | grep -v %%
f90 -c -o excite.o    --CollectFeedback                     excite.f90
../build_peak_none.0000/GemsFDTD > train.log 2>> train.err
f90 -c -o excite.o    --ApplyFeedback                     excite.f90
../build_peak_none.0000/GemsFDTD > train.log 2>> train.err
/usr/bin/postoptimizer
$ 

In the runspec command, we use the --fake option to quickly examine how the build will work. Using --fake is highly recommended when you are trying to debug your feedback commands, because it is much quicker than actually doing the builds. The fake output is searched with the Unix grep command; on Windows you would construct a findstr command. The example picks out lines of interest:

Having done all this, we can confirm that the above config file causes the build sequence to be compile - train - compile - train - othertool.

The sharp-eyed reader may wonder "What is this 'fdo_run2 = $command' line in the config file?" In the second pass, actually running the benchmark is optional, because it is possible that your pass 2 tools might not need another run of the benchmark. One could imagine a build system that notices everything interesting during the pass 1 run, and which causes that interesting information to be carried around by the binary itself, so that it never needs to be regenerated. To say "Yes, please do run it again", you add the above fdo_run2 line.

More complex examples can be constructed. If you mention PASS1, fdo_pre2, fdo_post2, PASS3, and fdo_post4, then you will end up with four passes. The numbers serve to order the passes, but if you skip numbers then the tools will not attempt to invent things to do in the "missing" passes.

VI.C. Interaction with the config file feedback option

If you use either PASSn* or fdo*n (or both), then, by default, feedback will happen. The config file option feedback provides an additional control, an "on/off" switch which allows, or disallows, the options selected with PASSn* and fdo*n.

A common usage model is to use PASSn* or fdo*n to turn on feedback for a whole set of benchmarks, and then use the option feedback=0 to turn it off for individual benchmarks. For example:

$ cat miriam1.cfg
action    = build
tune      = peak
runlist   = bzip2,mcf,libquantum

fdo_post1 = /usr/bin/merge_feedback fbdir

401.bzip2:
feedback=0

462.libquantum:
feedback=1

$ runspec --fake --config miriam1 | grep -e Building -e merge_feedback \
  | grep -v %%
  Building 401.bzip2 ref peak none default: (build_peak_none.0003)
  Building 429.mcf ref peak none default: (build_peak_none.0001)
/usr/bin/merge_feedback fbdir
  Building 462.libquantum ref peak none default: (build_peak_none.0000)
/usr/bin/merge_feedback fbdir
$ 

In this example, we build peak bzip2, mcf, and libquantum (notice the first three lines of the config file) and once again use --fake and grep in a fashion similar to the previous example. In this case, we can see that the fdo_post1 command was used for two of the three benchmarks from the runlist, since by default feedback=1.

VI.D. If the config file feedback option is used at multiple levels

If the feedback option is used at more than one level within a config file, the usual precedence rules apply. For example, the next config file specifies the feedback option both for a set of benchmarks and for some individual benchmarks:

$ cat miriam2.cfg
action    = build
tune      = peak
runlist   = bzip2,mcf,libquantum

fdo_post1 = /usr/bin/merge_feedback fbdir

default=peak:
feedback=0

401.bzip2:
feedback=0

462.libquantum:
feedback=1

$ runspec --fake --config miriam2 | grep -e Building -e merge_feedback \
   | grep -v %%
  Building 401.bzip2 ref peak none default: (build_peak_none.0003)
  Building 429.mcf ref peak none default: (build_peak_none.0001)
  Building 462.libquantum ref peak none default: (build_peak_none.0000)
/usr/bin/merge_feedback fbdir
$ 

In the above example, 429.mcf was built without feedback because of the feedback=0 line in the section named "default=peak:". But 462.libquantum was built with feedback (from fdo_post1) because it had a higher priority setting of feedback=1 in the section named "462.libquantum:".

VI.E. Interaction with runspec --feedback

As explained in section II.A, if an option is mentioned on both the command line and in a config file, the command line does not win over named sections, but it does win over the header section. Examples of both follow.

VI.F. More examples

For more examples of feedback, see the annotated example of a log file that uses FDO in section VIII.C. and the Example-medium.cfg config file in your config directory.


VII. The config file preprocessor

(History: as of the release of SPEC CPU2006 V1.0) The tools implement a configuration file macro preprocessor. The preprocessor can be used in a variety of ways; for example to communicate settings from your shell environment into runspec:

$ cat > jan.cfg
notes01 = Today, I am happily running in directory %{MYDIR} on system %{HOST}
$ runspec --config jan --define MYDIR=$PWD --define HOST=`hostname` \
  --fakereportable fp --output_format text | grep txt
        format: ASCII -> /jan/cpu2006/result/CFP2006.090.txt
$ grep Today /jan/cpu2006/result/CFP2006.090.txt
     Today, I am happily running in directory /jan/cpu2006/config on system civilized-03
$ 

The config file preprocessor is called a macro processor because it allows you to define macros, which are brief abbreviations for longer constructs. If you've ever used the C preprocessor, the concepts will be familiar, though the syntax is slightly different.

The preprocessor is automatically run whenever you use runspec. Or, you can run it separately, as configpp, which is documented in utility.html.

Preprocessor directives begin with the percent (%) character. This character must be the first character on the line. Any amount of spaces or tabs may separate the percent from the directive.

The following are okay:

%define foo
%    define bar
%		undef hello!

The following are not okay:

# Space in the first column
 %define foo

# Tab in the first column
	%define foo

# This isn't CPP!
#define foo

VII.A Defining macros

The preprocessor is all about macros. There are no macros defined by default, so unless you define some macros, the preprocessor can't do much for you.

Macros can be defined in two ways:

  1. On the command line, using the --define switch, or
  2. In the configuration file, using the '%define' preprocessor directive:
    # Define a simple macro
    %define foo bar
    # Now the macro called 'foo' has the value 'bar'
    
    # It's not necessary for a macro to have a value to be useful
    %define baz
    # Now the macro called 'baz' is defined, but it has no value.
    

    Note that no quoting is necessary when specifying the names of macros or their values.

Macros defined in both ways are entirely equivalent. Because ones set on the command-line are defined first, it's not possible to use the command line to override a macro definition that occurs in the config file itself. It may help to think of a series of '%define' directives, one per --define, as being prepended to the config file.

The values assigned to macros do NOT follow the same quoting rules as variables in the config file. In particular, you may NOT use line continuation, line appending, or block quotes. You may have a value of arbitrary length, but in the interests of config file readability and maintainability, please keep them relatively short.

You will receive a warning if a previously defined macro is re-defined.

Macro names may only be composed of alphanumeric characters, underscores, and hyphens, and they ARE case-sensitive.

A macro that has not been defined will not be substituted. A macro that has been defined, but which has not been assigned a value, has the value "1".

VII.B Un-doing macro definition

Sometimes you want to make the preprocessor forget about a macro that you taught it. This is easily accomplished.

Macros can be undefined in two ways:

  1. On the command line, using the --undef switch, or
  2. In the configuration file, using the '%undef' preprocessor directive.
    %define foo bar
    # Now the macro called 'foo' has the value 'bar'
    
    %undef foo
    # Now it doesn't
    

    Note that no quoting is necessary when specifying the names of macros.

Like macro definition, the undefinition requests can't affect macros set in the config file because those definitions effectively happen after the un-definition. For this reason, command-line undefinition is basically useless; it can only undo macros also set on the command line.
So why was such a useless ability added to the tools? The writer likes orthogonality.

VII.C Using macros

By now you're probably over that initial euphoric rush that comes from wantonly defining and undefining macros, and you're looking for something more. This is it!

When you want to use the value of a macro, you refer to it by name. Unfortunately, the syntax for this is not as simple as you might hope. It's not too complicated, though; to have the preprocessor expand the macro 'foo', just write

  %{foo}

in the place where you'd like it to appear. Given the following config file snippet:

%define foo Hello_
%define bar baz
%define foobar Huh?
%define foobaz What?
%define Hello_baz Please don't do this

Here's a handy table to see the various ways you can reference these values:

Macro reference Value
%{foo} Hello_
%{bar} baz
%{foobar} Huh?
%{foobaz} What?
%{Hello_baz} Please don't do this

Easy, right? The following is also possible:

%{foo%{bar}} What?
%{%{foo}%{bar}} Please don't do this

Because macro values can only be one line long, it's not possible to use the preprocessor to macro-ize large chunks of your config file at once, as may be common practice for advanced users of CPP.

A macro that has not been defined will not be substituted. Thus the following case is the expected behavior:

$ cat tmp.cfg
OPTIMIZE = %{FOO}
$ runspec -c tmp -i test -T base bzip
...
cc -c -o spec.o -DSPEC_CPU -DNDEBUG   %{FOO}   spec.c

Your C compiler probably won't know what to do with with %{FOO}. If you want to substitute an empty string, then assign it one:

$ runspec -c tmp -i test -T base --define FOO="" bzip

Note that the following are NOT equivalent:

   --define FOO=""
   --define FOO

The former sets FOO to an empty string. The latter, effectively, sets it to "1".

VII.D Conditionals

Defining, undefining, and expanding macros is quite an enjoyable activity in and of itself, and can even be useful on occasion. However, conditionals add an entirely new dimension to config file processing: the ability to include and exclude entire sections of text based on macros and their values.

VII.D.1. %ifdef .. %endif

The %ifdef conditional provides a way to determine whether or not a particular macro has been defined. If the named macro has been defined, the conditional is true, and the text to the matching %endif is included in the text of the config file as evaluated by runspec. Note that the matching %endif may not necessarily be the next %endif; conditionals may be nested.

For example, given the following section of a config file:

%define foo
%ifdef %{foo}
This text will be included
%endif

%ifdef %{bar}
This text will not be included
%endif

The preprocessor would produce the following output:

This text will be included

Note especially the quoting used for the macro names in the conditional; the only time macro name quoting may be omitted is when defining or undefining it.

VII.D.2. %ifndef .. %endif

The %ifndef conditional is the converse of %ifdef; If the named macro has not been defined, the conditional is true, and the text to the matching %endif is included in the text of the config file as evaluated by runspec. Note that the matching %endif may not necessarily be the next %endif; conditionals may be nested.

Given a slightly modified version of the example from earlier:

%define foo
%ifndef %{foo}
Now THIS text will not be included
%endif

%ifndef %{bar}
This text WILL be included
%endif

The preprocessor would produce the following output:

This text WILL be included

VII.D.3. %if .. %endif

Checking whether or not a macro is defined is quite useful, but it's just a subset of the more general conditional facility available. This general form is

%if expression
...
%endif

The expression is evaluated using a subset of the Perl interpreter, so the possibilities for testing values are fairly broad. For example,

%ifdef %{foo}
...
%endif

is exactly equivalent to

%if defined(%{foo})
...
%endif

Likewise,

%ifndef %{foo}
...
%endif

is exactly equivalent to

%if !defined(%{foo})
...
%endif

Using the general form, it's possible to string conditionals together:

%if defined(%{foo}) && !defined(%{bar}) || %{baz} == 0
...
%endif

If a macro contains a string value, you must supply quotes:

%if '%{foo}' eq 'Hello, Dave.'
...
%endif

You may also perform basic math on macro values:

%if %{foo} * 2004 > 3737
...
%endif

More precisely, the Perl operations allowed are the :base_core and :base_math bundles, with the ability to dereference and modify variables disallowed. For more details, see the source code for config.pl (the eval_pp_conditional subroutine) and Perl's own Opcode documentation.

VII.D.4. %else

It's possible to get by without the "else" part of the classic "if .. then .. else" trio, but it's not any fun. It works as you'd expect:

%define foo
%ifndef %{foo}
This text will not be included
%else
This text WILL be included (from the else clause)
%endif

The preprocessor would produce the following output:

This text WILL be included (from the else clause)

Only one %else per conditional is allowed.

VII.D.5. %elif

%elif is another convenience that's been added. For those not familiar with CPP, it's an "else if" construct. You may have as many of these as you'd like. Given:

%define foo Hello!

%if !defined(%{foo})
This text will not be included
%elif defined(%{bar})
This text won't be included either
%elif '%{foo}' eq 'Hello!'
This text WILL be included (from the second elif clause)
%else
Alas, the else here is left out as well.
%endif

The preprocessor would produce the following output:

This text WILL be included (from the second elif clause)

VII.E Informational Directives

It's often helpful to be able to warn or exit on certain conditions. Perhaps there's a macro that must be set to a particular value, or maybe it's just very highly recommended.

VII.E.1. %warning

%warning does just what you'd expect; when the preprocessor encounters this directive, it prints the text following to stdout and the current log file, along with its location within the file being read, and continues on.

Consider:

%if !defined(%{somewhat_important_macro})
% warning You have not defined somewhat_important_macro!
%endif

When run through the preprocessor, this yields the following output:

  $ configpp -c warning.cfg
  runspec v5127 - Copyright 1999-2007 Standard Performance Evaluation Corporation
  Using 'solaris-sparc' tools
  Reading MANIFEST... 18150 files
  Loading runspec modules................
  Locating benchmarks...found 31 benchmarks in 14 benchsets.
  Reading config file '/spec/cpu2006/config/warning.cfg'

  ****
  WARNING: You have not defined somewhat_important_macro! on line 2
           of /spec/cpu2006/config/warning.cfg
  ****

  Pre-processed configuration file dump follows:
  --------------------------------------------------------------------
  # Invocation command line:
  # /spec/cpu2006/bin/configpp -c warning.cfg
  # output_root was not used for this run
  ############################################################################

  --------------------------------------------------------------------

  The log for this run is in /spec/cpu2006/result/CPU2006.034.log

  runspec finished at Mon Apr 16 12:53:12 2007; 4 total seconds elapsed

VII.E.2. %error

Like %warning, %error logs an error to stderr and the log file. Unlike %warning, though, it then stops the run.

Consider a slightly modified version of the previous example:

%if !defined(%{REALLY_important_macro})
% error You have not defined REALLY_important_macro!
%endif

When run through the preprocessor, this yields the following output:

  $ configpp -c error.cfg
  runspec v5127 - Copyright 1999-2007 Standard Performance Evaluation Corporation
  Using 'solaris-sparc' tools
  Reading MANIFEST... 18150 files
  Loading runspec modules................
  Locating benchmarks...found 31 benchmarks in 14 benchsets.
  Reading config file '/spec/cpu2006/config/error.cfg'

  *************************************************************************
    ERROR: You have not defined REALLY_important_macro! on line 2
           of /spec/cpu2006/config/error.cfg
  *************************************************************************


  There is no log file for this run.

  runspec finished at Mon Apr 16 12:56:02 2007; 3 total seconds elapsed

  $ echo $?
  1

Unlike a warning, the error will be close to the last thing output. As you can see from the output of 'echo $?', runspec has exited with an error code 1.

VIII. Output files - and how they relate to your config file

This section describes how the location and contents of several kinds of output files are influenced by your config file.

VIII.A. Automatic backup of config files

It was mentioned above that the MD5 section of the config file is written automatically by the tools. Each time your config file is updated, a backup copy is made. Thus your config directory may soon come to look like this:

   $ cd $SPEC/config
   $ ls tmp.cfg*
   tmp.cfg                   tmp.cfg.2006-05-11_0842   tmp.cfg.2006-05-11_1631
   tmp.cfg.2006-05-10_2007   tmp.cfg.2006-05-11_0847   tmp.cfg.2006-05-11_1632
   tmp.cfg.2006-05-10_2010   tmp.cfg.2006-05-11_0853   tmp.cfg.2006-05-11_1731
   tmp.cfg.2006-05-10_2048   tmp.cfg.2006-05-11_0854   tmp.cfg.2006-05-11_1731a
   tmp.cfg.2006-05-10_2051   tmp.cfg.2006-05-11_0855   tmp.cfg.2006-05-12_0921
   tmp.cfg.2006-05-10_2054   tmp.cfg.2006-05-11_0856   tmp.cfg.2006-05-13_0846
   tmp.cfg.2006-05-10_2058   tmp.cfg.2006-05-11_0857   tmp.cfg.2006-05-13_0846a
   tmp.cfg.2006-05-10_2105   tmp.cfg.2006-05-11_0858   tmp.cfg.2006-05-13_0849
   tmp.cfg.2006-05-10_2105a  tmp.cfg.2006-05-11_0903   tmp.cfg.2006-05-13_0850
   tmp.cfg.2006-05-10_2106   tmp.cfg.2006-05-11_0904   tmp.cfg.2006-05-16_0957
   tmp.cfg.2006-05-10_2125   tmp.cfg.2006-05-11_0905   tmp.cfg.2006-05-18_1133
   tmp.cfg.2006-05-10_2125a  tmp.cfg.2006-05-11_0906   tmp.cfg.2006-05-19_1626
   tmp.cfg.2006-05-10_2126   tmp.cfg.2006-05-11_1348   tmp.cfg.2006-05-19_1627
   tmp.cfg.2006-05-10_2127   tmp.cfg.2006-05-11_1349   tmp.cfg.2006-05-19_1634
   tmp.cfg.2006-05-11_0811   tmp.cfg.2006-05-11_1349a  tmp.cfg.2006-05-19_1638
   tmp.cfg.2006-05-11_0823   tmp.cfg.2006-05-11_1349b  tmp.cfg.2006-05-19_1718
   tmp.cfg.2006-05-11_0835   tmp.cfg.2006-05-11_1553   tmp.cfg.2006-05-19_1720
   tmp.cfg.2006-05-11_0836   tmp.cfg.2006-05-11_1556   tmp.cfg.2006-05-19_1731
   tmp.cfg.2006-05-11_0838   tmp.cfg.2006-05-11_1557   tmp.cfg.2006-05-21_0611
   tmp.cfg.2006-05-11_0839   tmp.cfg.2006-05-11_1627   tmp.cfg.2006-05-21_0622
   tmp.cfg.2006-05-11_0840   tmp.cfg.2006-05-11_1629   tmp.cfg.2006-05-21_0652
   $

If this feels like too much clutter, you can disable the backup mechanism, as described under backup_config. Note that doing so may leave you with a risk of losing the config file in case of a filesystem overflow or system crash. A better idea may be to periodically remove just portions of the clutter, for example by typing:

   $ rm tmp.cfg.2006-05-1*
   $ ls tmp.cfg*
   tmp.cfg                  tmp.cfg.2006-05-21_0622
   tmp.cfg.2006-05-21_0611  tmp.cfg.2006-05-21_0652
   $ 

VIII.B. The log file and verbosity levels

$SPEC/result (Unix) or %SPEC%\result (Windows) contains reports and log files. When you are doing a build, you will probably find that you want to pay close attention to the log files such as CPU2006.001.log. Depending on the verbosity level that you have selected, it will contain detailed information about how your build went.

The CPU2006 tool suite provides for varying amounts of output about its actions during a run. These levels range from the bare minimum of output (level 0) to copious streams of information that are probably useful only to tools developers (level 99). Selecting one output level gives you the output from all lower levels, which may cause you to wade through more output than you might like.

VIII.B.1. Useful Search Strings

When you are trying to find your way through a log file, you will probably find these (case-sensitive) search strings useful:

runspec:The runspec command for this log.
RunningPrinted at the top of a run of a benchmark.
#Printed at the top of a run of a benchmark for runs with multiple iterations. Useful for finding the ref workloads in reportable runs.
runtimePrinted at the end of a benchmark run.
CopyTimes for individual copies in a SPECrate run.
BuildingPrinted at the beginning of a benchmark compile.
Elapsed compilePrinted at the end of a benchmark compile.

VIII.B.2. About Temporary Debug Logs

There are also temporary debug logs, such as CPU2006.001.log.debug. A debug log contains very detailed debugging output from the SPEC tools, as if --verbose 99 had been specified.

For a successful run, the debug log will be removed automatically, unless you specify "--keeptmp" on the command line, or "keeptmp=1" in your config file.

For a failed run, the debug log is kept. The debug log may seem overwhelmingly wordy, repetitive, detailed, redundant, repetitive, and long-winded, and therefore useless. Suggestion: after a failure, try looking in the regular log first, which has a default verbosity level of 5. If your regular log doesn't have as much detail as you wish, then you can examine the additional detail in the debug log.

If you file a support request, you may be asked to send in the debug log.

(History: Temporary debug logs were added in SPEC CPU2006 V1.1.)

VIII.B.3. Definitions of verbosity levels

The 'level' referred to in the table below is selected either in the config file verbose option or in the runspec command as in 'runspec --verbose n'.

Levels higher than 99 are special; they are always output to your log file. You can also see them on the screen if you set verbosity to the specified level minus 100. For example, the default log level is 5. This means that on your screen you will get messages at levels 0 through 5, and 100 through 105. In your log file, you'll find find the same messages, plus the messages at levels 106 through 199.

Level What you get
0 Basic status information, and most errors. These messages can not be turned off.
1 List of the benchmarks which will be acted upon.
2 A list of possible output formats, as well as notification when beginning and ending each phase of operation (build, setup, run, reporting).
3 A list of each action performed during each phase of operation (e.g. "Building 176.gcc", "Setting up 253.perlbmk")
4 Notification of benchmarks excluded
5 (default) Notification if a benchmark somehow was built but nevertheless is not executable.
6 Time spent doing automatic flag reporting.
7 Actions to update SPEC-supplied flags files.
10 Information on basepeak operation.
12 Errors during discovery of benchmarks and output formats.
15 Information about certain updates to stored config files
24 Notification of additions to and replacements in the list of benchmarks.
30 A list of options which are included in the MD5 hash of options used to determine whether or not a given binary needs to be recompiled.
35 A list of key=value pairs that can be used in command and notes substitutions, and results of env_var settings.
40 A list of 'submit' commands for each benchmark.
70 Information on selection of median results.
89 Progress comparing run directory MD5s for executables.
90 Time required for various internal functions in the tools.
95, 96, 97, 98 Flag parsing progress during flag reporting (progressively more detail)
99 Gruesome detail of comparing MD5 hashes of files being copied during run directory setup.
--- Messages at the following levels will always appear in your log files ---
100 Various config file errors, such as bad preprocessor directives, bad placement of certain options, illegal characters...
102 Information about output formats that could not be loaded.
103 A tally of successes and failures during the run broken down by benchmark.
106 A list of runtime and calculated ratio for each benchmark run.
107 Dividers to visually block each phase of the run.
110 Elapsed time for each portion of a workload (if an executable is invoked more than once).
120 Messages about which commands are being issued for which benchmarks.
125 A listing of each individual child processes' start, end, and elapsed times.
130 A nice header with the time of the runspec invocation and the command line used. Information about what happened with your sysinfo program
140 General information about the settings for the current run.
145 Messages about file comparisons.
150 List of commands that will be run, and details about the settings used for comparing output files. Also the contents of the makefile written.
155 Start, end, and elapsed times for benchmark run.
160 Start, end, and elapsed times for benchmark compilation.
180 stdout and stderr from commands run
190 Start and stop of delays
191 Notification of command line used to run specinvoke.

VIII.C. Log file example: Feedback-directed optimization.

In this section, two things are attempted: a guided tour of how to find some of the interesting parts in a log file, and an explanation of how the SPEC toolset implements feedback-directed optimization, which is commonly abbreviated FDO. The technique is also known as Profile-Based Optimization, or PBO.

To use FDO, you typically compile a program twice. The first compile creates an image with instrumentation. Then, you run the program, with a "training" workload, and the instrumentation collects data about the run: a profile. Finally, you re-compile the program, and the compiler uses the profile to improve its optimizations.

SPEC CPU2006 makes all of this relatively easy. Here's a config file that builds 401.bzip2 with FDO:

   $ cat mat.cfg
   ext                  = blue271
   iterations           = 1
   output_format        = text
   teeout               = 1
   runlist              = bzip2
   tune                 = peak

   int=peak:
   OPTIMIZE       = -O
   PASS1_CFLAGS   = -xprofile=collect:./fb
   PASS1_LDFLAGS  = -xprofile=collect:./fb
   PASS2_CFLAGS   = -xprofile=use:./fb
   PASS2_LDFLAGS  = -xprofile=use:./fb
   $ 

The PASSxx lines above cause FDO to happen. Each of the profiling switches is specified twice because we need them to be applied both for compiles and for the link. Let's invoke runspec with the above config file, searching the output for lines that contain either "Training" or "bzip2.o", which is a handy string to pick out both a sample compile line and the link line:

   $ runspec --config mat --size test | grep -e Training -e bzip2.o
   cc -c -o bzip2.o -DSPEC_CPU -DNDEBUG    -xprofile=collect:./fb 
      -O bzip2.c
   cc -O -xprofile=collect:./fb  spec.o blocksort.o bzip2.o bzlib.o 
      compress.o crctable.o decompress.o huffman.o randtable.o 
      -o bzip2
   Training 401.bzip2
   cc -c -o bzip2.o -DSPEC_CPU -DNDEBUG    -xprofile=use:./fb    
      -O bzip2.c
   cc -O -xprofile=use:./fb  spec.o blocksort.o bzip2.o bzlib.o 
      compress.o crctable.o decompress.o huffman.o randtable.o 
      -o bzip2
   $ 

Above, you can see the basic flow: compile using the switch -xprofile=collect for both the compile and link lines; run the training workload; then recompile with -xprofile=use.

Let's go a little deeper by taking apart the log file. This section uses the actual log file from the above runspec command, but white space has been adjusted.

The first thing to look for when you're trying to make sure you've found the right log file is the line that contains the string runspec:

   $ grep runspec: *079.log
   runspec: runspec --config mat --size test
   $ 

Yes, this looks like the right log. To find the section where the benchmark is built, search for "Building", which is soon followed by information about what was written to the makefile:

------------------------------------------------------------------------
When checking options for /vampir/benchspec/CPU2006/401.bzip2/exe/bzip2_peak.blue271, 
  no MD5 sums were
  found in the config file.  They will be installed after build.
  Building 401.bzip2 test peak blue271 default: (build_peak_blue271.0000)
Wrote to makefile 
'/vampir/benchspec/CPU2006/401.bzip2/run/build_peak_blue271.0000/Makefile.deps':

# These are the build dependencies

# End dependencies
Wrote to makefile 
'/vampir/benchspec/CPU2006/401.bzip2/run/build_peak_blue271.0000/Makefile.spec':
TUNE=peak
EXT=blue271
NUMBER=401
NAME=bzip2
SOURCES= spec.c blocksort.c bzip2.c bzlib.c compress.c crctable.c \
	 decompress.c huffman.c randtable.c
EXEBASE=bzip2
NEED_MATH=
BENCHLANG=C
ONESTEP=
CONESTEP=

OPTIMIZE         = -O
OS               = unix
PASS1_CFLAGS     = -xprofile=collect:./fb
PASS1_LDFLAGS    = -xprofile=collect:./fb
PASS2_CFLAGS     = -xprofile=use:./fb
PASS2_LDFLAGS    = -xprofile=use:./fb
abstol           = 

To tell the tools that we want to use FDO, we set PASS1_<language>FLAGS and PASS2_<language>FLAGS. If the tools see any use of these flags, they will perform two compiles. The particular compiler used in this example expects to be invoked twice: once with -xprofile=collect:./fb and then again with -xprofile=use:./fb.

A useful search string in the log is "specmake":

worklist         = list
  Compile for '401.bzip2' started at: Mon May 22 21:31:38 2006 (1148347898)
Issuing make.clean command 'specmake clean'
Start make.clean command: Mon May 22 21:31:38 2006 (1148347898)
rm -rf bzip2 bzip2.exe *.o  *.fppized.f*
find . \( -name \*.o -o -name '*.fppized.f*' \) -print | xargs rm -rf
rm -rf core 
Stop make.clean command: Mon May 22 21:31:41 2006 (1148347901)
Elapsed time for make.clean command: 00:00:03 (3)
Issuing fdo_make_pass1 command 'specmake build FDO=PASS1'
Start fdo_make_pass1 command: Mon May 22 21:31:45 2006 (1148347905)
cc -c -o spec.o -DSPEC_CPU -DNDEBUG  -xprofile=collect:./fb    -O  spec.c
cc -c -o blocksort.o -DSPEC_CPU -DNDEBUG  -xprofile=collect:./fb    -O  blocksort.c
cc -c -o bzip2.o -DSPEC_CPU -DNDEBUG  -xprofile=collect:./fb    -O  bzip2.c
cc -c -o bzlib.o -DSPEC_CPU -DNDEBUG  -xprofile=collect:./fb    -O  bzlib.c
"bzlib.c", line 911: warning: statement not reached
"bzlib.c", line 1288: warning: statement not reached
cc -c -o compress.o -DSPEC_CPU -DNDEBUG  -xprofile=collect:./fb    -O  compress.c
cc -c -o crctable.o -DSPEC_CPU -DNDEBUG  -xprofile=collect:./fb    -O  crctable.c
cc -c -o decompress.o -DSPEC_CPU -DNDEBUG  -xprofile=collect:./fb    -O  decompress.c
cc -c -o huffman.o -DSPEC_CPU -DNDEBUG  -xprofile=collect:./fb    -O  huffman.c
cc -c -o randtable.o -DSPEC_CPU -DNDEBUG  -xprofile=collect:./fb    -O  randtable.c
cc  -O         -xprofile=collect:./fb  spec.o blocksort.o bzip2.o bzlib.o compress.o 
   crctable.o decompress.o huffman.o randtable.o                     -o bzip2

You can see above that specmake is invoked with FDO=PASS1, which causes the switches from PASS1_CFLAGS to be used. If you want to understand exactly how this affects the build, read $SPEC/benchspec/Makefile.defaults, along with the document $SPEC/Docs/makevars.html.

To find the training run, search forward for Training

O: OPTIMIZE="-O"
O: PASS1_LDFLAGS="-xprofile=collect:./fb"
C: LDOUT="-o options"
Stop options1 command: Mon May 22 21:31:57 2006 (1148347917)
Elapsed time for options1 command: 00:00:02 (2)
Training 401.bzip2
Commands to run: 
  -C /vampir/benchspec/CPU2006/401.bzip2/run/build_peak_blue271.0000
  -o input.program.out -e input.program.err ../build_peak_blue271.0000/bzip2 input.program 10
  -o byoudoin.jpg.out -e byoudoin.jpg.err ../build_peak_blue271.0000/bzip2 byoudoin.jpg 5
  -o input.combined.out -e input.combined.err ../build_peak_blue271.0000/bzip2 input.combined 80
Specinvoke: /vampir/kit91/bin/specinvoke 
    -d /vampir/benchspec/CPU2006/401.bzip2/run/build_peak_blue271.0000 
    -e speccmds.err -o speccmds.stdout -f speccmds.cmd -C
Issuing command '/spec/john/kit91/bin/specinvoke 
    -d /vampir/benchspec/CPU2006/401.bzip2/run/build_peak_blue271.0000 
    -e speccmds.err -o speccmds.stdout -f speccmds.cmd -C'
Start command: Mon May 22 21:31:58 2006 (1148347918)
Stop command: Mon May 22 21:39:57 2006 (1148348397)
Elapsed time for command: 00:07:59 (479)
Workload elapsed time (0:1) = 52.091489 seconds
Workload elapsed time (0:2) = 73.627057 seconds
Workload elapsed time (0:3) = 345.535733 seconds
copy 0 finished @ Mon May 22 21:39:49 2006.  Total elapsed time: 471.254279
comparing files in '/vampir/benchspec/CPU2006/401.bzip2/run/build_peak_blue271.0000'
comparing 'byoudoin.jpg.out' with abstol=, binary=, calctol=0, cw=, floatcompare=, 
   ignorecase=, obiwan=, reltol=, skipabstol=, skipobiwan=, skipreltol=, skiptol=

The key lines to notice above are the ones just after "Commands to run:", which begin with -o. These lines cause specinvoke to run the freshly-built bzip2 three times. The -o and -e parameters to specinvoke indicate where it is to send standard output and standard error. Thus, for example

-o byoudoin.jpg.out -e byoudoin.jpg.err ../build_peak_blue271.0000/bzip2 byoudoin.jpg 5

will cause this command to actually be run:

../build_peak_blue271.0000/bzip2 byoudoin.jpg 5 > byoudoin.jpg.out 2> byoudoin.jpg.err

For more information on specinvoke, see utility.html.

For this particular benchmark, SPEC has supplied three workloads that are used to train bzip2. Why three workloads? Because a training set should provide the compiler with information about usage of the program which is, in the opinion of the developer, representative of real-world use. Of course, if the developer has actual evidence, instead of merely an opinion, so much the better! In this instance, the developer of the training workload has chosen to train bzip with:

  1. The file input.program, which is a program binary;
  2. An input in .jpg format, which is inherently already compressed and therefore not a particularly useful application of bzip2 -- but real users do this; and
  3. An input that contains a mixture of highly compressible and not-very-compressible data: input.combined.

You can see the inputs that SPEC has provided for training purposes in the directories nnn.benchmark/data/train/input and nnn.benchmark/data/all/input. In some cases, the training workloads required significant development effort, but as a user of the suite you don't have to worry about that; you can simply apply them. SPEC is aware that there is some variation in the fidelity between the training workloads vs. the timed "ref" workloads. In the real world, also, training workloads used by program developers do not correspond perfectly to how end users apply the programs.

In any case, the tester who employs the SPEC CPU2006 suite does not have to come up with his or her own training workloads, and, indeed, is not allowed to do so under the run rules.

Notice that the log file tells us both the time for the individual invocations of bzip2: 52.09, 73.63, and 345.54 seconds; and the total time: 471.25 seconds.

Finally, the compiler is run a second time, to use the profile feedback and build a new executable, at the second specmake build:

Start fdo_make_clean_pass2 command: Mon May 22 21:40:02 2006 (1148348402)
rm -rf bzip2 bzip2.exe *.o  *.fppized.f*
find . \( -name \*.o -o -name '*.fppized.f*' \) -print | xargs rm -rf
Stop fdo_make_clean_pass2 command: Mon May 22 21:40:05 2006 (1148348405)
Elapsed time for fdo_make_clean_pass2 command: 00:00:03 (3)
Issuing fdo_make_pass2 command 'specmake build FDO=PASS2'
Start fdo_make_pass2 command: Mon May 22 21:40:05 2006 (1148348405)
cc -c -o spec.o -DSPEC_CPU -DNDEBUG    -xprofile=use:./fb    -O   spec.c
cc -c -o blocksort.o -DSPEC_CPU -DNDEBUG    -xprofile=use:./fb    -O   blocksort.c
cc -c -o bzip2.o -DSPEC_CPU -DNDEBUG    -xprofile=use:./fb    -O   bzip2.c
cc -c -o bzlib.o -DSPEC_CPU -DNDEBUG    -xprofile=use:./fb    -O   bzlib.c
"bzlib.c", line 911: warning: statement not reached
"bzlib.c", line 1288: warning: statement not reached
cc -c -o compress.o -DSPEC_CPU -DNDEBUG    -xprofile=use:./fb    -O   compress.c
cc -c -o crctable.o -DSPEC_CPU -DNDEBUG    -xprofile=use:./fb    -O   crctable.c
cc -c -o decompress.o -DSPEC_CPU -DNDEBUG    -xprofile=use:./fb    -O   decompress.c
cc -c -o huffman.o -DSPEC_CPU -DNDEBUG    -xprofile=use:./fb    -O   huffman.c
cc -c -o randtable.o -DSPEC_CPU -DNDEBUG    -xprofile=use:./fb    -O   randtable.c
cc  -O         -xprofile=use:./fb  spec.o blocksort.o bzip2.o bzlib.o compress.o 
   crctable.o decompress.o huffman.o randtable.o                     -o bzip2
Stop fdo_make_pass2 command: Mon May 22 21:40:15 2006 (1148348415)
Elapsed time for fdo_make_pass2 command: 00:00:10 (10)
Issuing options2 command 'specmake options FDO=PASS2'
Start options2 command: Mon May 22 21:40:15 2006 (1148348415)

This time, specmake is invoked with FDO=PASS2, which is why the compile picks up the PASS2_CFLAGS.

And that's it. The tools did most of the work; the user simply set the PASSn flags in the config file.

VIII.D. Help, I've got too many logs!

If you do a very large number of builds and runs, you may find that your result directory gets far too cluttered. If it does, you should feel free to issue commands such as these on Unix systems:

   cd $SPEC
   mv result result_old
   mkdir result

On Windows, you could say:

   cd %SPEC%
   rename result result_old
   mkdir result

VIII.E. Finding the build directory

As described under "About Disk Usage" in runspec.html, the CPU2006 tools do the actual builds and runs in newly created directories. The benchmark sources are never modified in the src directory.

The build directories for a benchmark are located underneath that benchmarks' top-level directory, typically $SPEC/benchspec/CPU2006/nnn.benchmark/build (Unix) or %SPEC%\benchspec\CPU2006\nnn.benchmark\build (Windows).

(If you are using the output_root feature, then the first part of that path will change to be your requested root instead of SPEC; and if you change the default for build_in_builddir, then the last part of that path will be run instead of build.)

(History: As of the release of SPEC CPU2006 V1.0) Tthe build directories have logical names, typically of the form build_<tune>_<extension>.0000. For example, after the command runspec --config jun09a --action build --tune base libquantum, the following directory would be created:

$ cd $SPEC/benchspec/CPU2006/462.libquantum/build
$ pwd
/spec/joelw/benchspec/CPU2006/462.libquantum/build
$ ls -ld build*jun*
drwxrwxr-x   2 joelw ptg    1536 Jun 10 14:49 build_base_jun09a.0000
$ 

On Windows, you would say cd %SPEC%\benchspec\CPU2006\462.libquantum\build followed by dir build*.

If the directory build_<tune>_<extension>.0000 already exists when a new build is attempted for the same tuning and extension, the directory will be re-used, unless:

In such cases, the 0000 will be incremented until a name is generated that is available. You can find locked directories by searching for lock=1 in the file $SPEC/benchspec/CPU2006/<nnn.benchmark>/run/list (Unix) or %SPEC%\benchspec\CPU2006\<nnn.benchmark>\run\list (Windows).

When more than one build directory has been created for a given tuning and extension, you may need to trace the directory back to the specific build attempt that created it. You can do so by searching for the directory name in the log files:

$ grep Building *log | grep build_base_jun09a.0001
CPU2006.380.log:  Building 462.libquantum ref base jun09a default: (build_base_jun09a.0001)
$ 

In the above example, the grep command locates log #380 as the log that corresponds to this run directory. On Windows, of course, you would use findstr instead of grep.

VIII.F. Files in the build directory

A variety of files are output to the build directory. Here are some of the key files which can usefully be examined:

Makefile.spec The components for make that were generated for the current config file with the current set of runspec options.
options.out For 1 pass compile: build options summary.
options1.out For N pass compile: summary of first pass.
options2.out For N pass compile: summary of second pass.
make.out For 1 pass compile: detailed commands generated.
fdo_make_pass1.out For N pass compile: detailed commands generated for 1st pass.
fdo_make_pass2.out For N pass compile: detailed commands generated for 2nd pass.
*.err The output from standard error corresponding to the above files.

VIII.G. For more information

For more information about how the run directories work, see the descriptions of specinvoke, specmake, and specdiff in utility.html.

IX. About Alternate Sources

Sometimes a portability issue may require use of alternative source code for a benchmark, and SPEC may issue the alternate as a "src.alt". The effect of applying a src.alt is to modify the sources in the build directory. This chapter provides an example of applying a src.alt, and introduces development of src.alts.

IX.A. Example: Applying a src.alt

This example shows the effect of a src.alt, 403.gcc.no_line.cpu2006.v1.2.tar.xz, which is an example src.alt. There are not yet any approved for SPEC CPU2006 V1.2; this is just an example. A src.alt must have a README, which for this example says, in part:

This change reduces confusion about line
numbers in compiler error or warning messages.

The diff is quite large, but the change is easy to describe.  All of the 
'#line' preprocessor directives in generated sources are removed.

If it were a real src.alt, one would download it from www.spec.org/cpu2006/src.alt, and then apply it per the instructions (found in the same README from which the above paragraphs were excerpted):

$ . ./shrc
$ specxz -dc 403.gcc.no_line.cpu2006.v1.2.tar.xz | spectar -xf  -

Let's set up two build directories - one without the change:

$ cat $SPEC/config/without.cfg
ext          = without
$

$ runspec --action buildsetup -T base -c without -v 1 403.gcc
runspec v6624 - Copyright 1999-2011 Standard Performance Evaluation Corporation
Using 'macosx' tools
Reading MANIFEST... 19145 files
Loading runspec modules................
Locating benchmarks...Reading config file '/Volumes/CPU2006/cpu2006/config/without.cfg'
Benchmarks selected: 403.gcc


The log for this run is in /Volumes/CPU2006/cpu2006/result/CPU2006.009.log

runspec finished at Thu Jul 21 17:20:53 2011; 2 total seconds elapsed
$ 

And a build directory with the change:

$ cat $SPEC/config/with.cfg
ext          = with
403.gcc:
srcalt      = no_line
$ 

$ runspec --action buildsetup -T base -c with -v 1 403.gcc
runspec v6624 - Copyright 1999-2011 Standard Performance Evaluation Corporation
Using 'macosx' tools
Reading MANIFEST... 19145 files
Loading runspec modules................
Locating benchmarks...Reading config file '/Volumes/CPU2006/cpu2006/config/with.cfg'
Benchmarks selected: 403.gcc
403.gcc (base): "no_line" src.alt was used.


The log for this run is in /Volumes/CPU2006/cpu2006/result/CPU2006.010.log

runspec finished at Thu Jul 21 17:22:11 2011; 2 total seconds elapsed
$

If we go look at those directories, the "with" directory has the expected change from the src.alt. As mentioned in the README above, preprocessor directives are removed (lines marked "-"). No lines are added, but if they were, they would be marked with "+":

$ diff -u build_base_without.0000/c-parse.c build_base_with.0000/c-parse.c
--- build_base_without.0000/c-parse.c   2007-10-09 23:55:31.000000000 +0100
+++ build_base_with.0000/c-parse.c      2011-07-21 17:22:11.000000000 +0100
@@ -71,8 +71,6 @@
 #define        CLASS   321
 #define        ALIAS   322
 
-/* #line 34 "c-parse.y" */
-
 #include "config.h"
 #include "system.h"
 #include "tree.h"
@@ -102,10 +100,8 @@
 /* Rename the "yyparse" function so that we can override it elsewhere.  */
 #define yyparse yyparse_1
 
-/* #line 67 "c-parse.y" */
 typedef union {long itype; tree ttype; enum tree_code code;
  const char *filename; int lineno; } YYSTYPE;
-/* #line 200 "c-parse.y" */
 
 /* Number of statements (loosely speaking) and compound statements 
    seen so far.  */
$

IX.B. Developing a src.alt (brief introduction)

If you are trying to create a new alternative source, you should become familiar with how to work in a sandbox, temporarily abandoning the tools during your development phase. (Or, you can use convert_to_development to make the whole installation into one big sandbox.) Once you have developed your alternative source, you'll want to package it up with makesrcalt, and you'll need to contact SPEC to get the source approved. (History: The convert_to_development utility was added with SPEC CPU2006 V1.1, and makesrcalt was first newly documented in SPEC CPU2006 V1.1.) Both are much more fully discussed in utility.html.

X. Troubleshooting

When something goes wrong, here are some things to check:

  1. Are there any obvious clues in the log file? Search for the word "Building". Keep searching until you hit the next benchmark AFTER the one that you are interested in. Now scroll backward one screen's worth of text.

  2. Did your desired switches get applied? Go to the build directory, and look at options*out.

  3. Did the tools or your compilers report any errors? Look in the build directory at *err.

  4. What happens if you try the build by hand? See the section on specmake in utility.html.

  5. If an actual run fails, what happens if you invoke the run by hand? See the information about "specinvoke -n" in utility.html

  6. Do you understand what is in your path, and why? Sometimes confusion can be greatly reduced by ensuring that you have only what is needed, avoiding, in particular, experimental and non-standard versions of standard utilities.

    Note: on Windows systems, SPEC recommends that Windows/Unix compatibility products should be removed from the %PATH% prior to invoking runspec, in order to reduce the probability of certain difficult-to-diagnose error messages.

  7. Try asking the tools to leave more clues behind, with keeptmp.


Copyright 1999-2011 Standard Performance Evaluation Corporation
All Rights Reserved