- 1 - 1. _G_e_n_e_r_a_l 1.1 _C_l_a_s_s_i_f_i_c_a_t_i_o_n 057.SDET is a System level benchmark, that is designed to measure the system performance of any computer system. 1.2 _D_e_s_c_r_i_p_t_i_o_n The System performance broadly includes the following hardware and software components: +o CPU performance, floating point speed, compiler efficiency, cache size and clock speed, +o Operating system ( system calls & how fast they are executed), +o System bus bandwidth, +o I/O subsystem (bus bandwidth, latency, and throughput), +o Various I/O control mechanisms (IOP's), and +o Disk/ tty/ network subsystems. The benchmark contains three main functional parts: 1. The Internal Driver, 2. The Term Scripts, and 3. The Clone Directories. The benchmark is coded to measure the execution time of each child process and give the average time as an optional output along with the total wall clock time obtained through the SPEC generic Makefile. 1. The Internal Driver: These benchmarks use an internal driver (a C program "drvr.exec.c" in the "tools" directory) which spawns (forks) a optional number of child processes, each of which executes a specific commands script. For example the child process 5 executes the "term.5" script available in the "scripts" directory. - 2 - 2. The Term Scripts: A term script emulates a user sitting at a terminal typing commands at the baud rate of line connected to a system. As the aim of the benchmark is to find the throughput of the system when several processes are being executed concurrently, each forked off process executes a term script from the "scripts" directory. Each term script is a random collection of UNIX commands. The frequency and the total number of UNIX commands is the same for all the term scripts. The term scripts are freshly created everytime after the "make clobber" command is executed. The maximum number of term scripts that can be optioned is 999. 3. The Clone Directories: A master clone directory is supplied with the benchmark which will be multiplicated, depending on the number of child processes optioned. These directories can be optionally created on specific partitions requested by the user ( to be specified in the M.vendor file). If no partitions are given, then all the clone directories are created under "clones" in the benchmark directory. 1.3 _S_o_u_r_c_e_/_A_u_t_h_o_r The 057.SDET benchmark is sponsored by the AT&T Computer Systems Division of the AT&T Bell laboratories. 1.4 _V_e_r_s_i_o_n_/_D_a_t_e The current version is dated: May 14, 1991 1.5 _J_u_s_t_i_f_i_c_a_t_i_o_n__-__A_d_d_e_d__V_a_l_u_e__t_o__C_u_r_r_e_n_t__R_e_l_e_a_s_e Currently SPEC release 1 suite of benchmarks measure the speed metric of a system, thus evaluating the CPU performance along with floating point speed, compiler efficiency, cache size and clock rate. As explained above, the system level benchmarks evaluate many other software and hardware components of a system. The throughput metric obtained from the SDET and SDEF benchmarks give a more - 3 - realistic evaluation of a system for "software development environment" workloads. Hence these benchmarks definitely add value and depth to the release 2 benchmark suite. 1.6 _A_d_d_i_t_i_o_n_a_l__G_e_n_e_r_a_l__I_n_f_o_r_m_a_t_i_o_n Additional information is available in the literature referenced in the "articles" list. Also refer to user.mm document included in the "docs" directory. 2. _P_e_r_f_o_r_m_a_n_c_e 2.1 _M_e_t_r_i_c The throughput metric is measured in "SCRIPTS PER HOUR". This metric is calculated by the benchmark tools included in the bin and a throughput metric is displayed for each run on the terminal, as well as stored in the result directory. 2.2 _E_l_a_p_s_e_d__T_i_m_e Elapsed time is measured using the tool "time" given in the "bin" directory, which transforms the one lin "/bin/time" output in BSD flavoured UNIX's, to standard SYSTEM V three line output. 2.3 _R_e_p_o_r_t_s 2.4 _A_d_d_i_t_i_o_n_a_l__P_e_r_f_o_r_m_a_n_c_e__I_n_f_o_r_m_a_t_i_o_n 3. _S_o_f_t_w_a_r_e 3.1 _L_a_n_g_u_a_g_e The benchmark programs are written in C language and the shell programs are written in Bourne shell. 3.2 _O_p_e_r_a_t_i_n_g__S_y_s_t_e_m UNIX(R) System 5 any version including release 4.0 and above and all Berkeley BSD versions. - 4 - 3.3 _P_o_r_t_a_b_i_l_i_t_y Currently portability is tested on SPEC vendor member machines only. 3.4 _V_e_c_t_o_r_i_z_a_b_i_l_i_t_y_/_M_u_l_t_i_-_p_r_o_c_e_s_s_o_r__I_s_s_u_e_s Not an issue. 3.5 _M_i_s_c_e_l_l_a_n_e_o_u_s__S_o_f_t_w_a_r_e The benchmark workload needs an "nroff" package which is included in "DWB" pacakge optionally available with UNIX operating system. 3.6 _K_n_o_w_n__B_u_g_s None. 3.7 _A_d_d_i_t_i_o_n_a_l__S_o_f_t_w_a_r_e__C_o_n_s_i_d_e_r_a_t_i_o_n_s 4. _H_a_r_d_w_a_r_e 4.1 _M_e_m_o_r_y No constraints 4.2 _D_i_s_k_s No constraints. Please read the documentation on TUNE and DBUG. 4.3 _C_o_m_m_u_n_i_c_a_t_i_o_n No constraints. 4.4 _S_p_e_c_i_a_l__H_a_r_d_w_a_r_e None. 4.5 _A_d_d_i_t_i_o_n_a_l__H_a_r_d_w_a_r_e__C_o_n_s_i_d_e_r_a_t_i_o_n_s None. - 5 - 5. _O_p_e_r_a_t_i_o_n_a_l 5.1 _D_i_s_k__S_p_a_c_e Currently the benchmark needs approximately 1650 K bytes of space, which does not include the space for clone directories. Each clone directory needs 917 Kbytes of additional memory. 5.2 _I_n_s_t_a_l_l_a_t_i_o_n Available with the SPEC release 2 tape and needs no special instructions for installation. 5.3 _E_x_e_c_u_t_i_o_n "make -f M.xxx" runs a single copy of the benchmark for testing purposes only. "make -f M.xxx runsequence" runs a sequence of runs as specified in the M.xxx wrapper makefile. 5.4 _C_o_r_r_e_c_t_n_e_s_s__V_e_r_i_f_i_c_a_t_i_o_n The output generated from each script (clone process) is stored in run.x files in the output and is compared with a generic file output after sorting. This is done automatically after each run and if there are failures, the subsequent runs of the benchmark are aborted. Result is not calculated in case of errors. 5.5 _A_d_d_i_t_i_o_n_a_l__O_p_e_r_a_t_i_o_n_a_l__C_o_n_s_i_d_e_r_a_t_i_o_n_s 5.6 _S_a_m_p_l_e__R_u_n The terminal output for a sample run is included here.