SPEC(R) CPU2017 Integer Rate Result Hewlett Packard Enterprise ProLiant DL385 Gen10 (2.50 GHz, AMD EPYC 7261) Test Sponsor: HPE CPU2017 License: 3 Test date: Jun-2019 Test sponsor: HPE Hardware availability: Feb-2019 Tested by: HPE Software availability: Dec-2018 Base Base Base Peak Peak Peak Benchmarks Copies Run Time Rate Copies Run Time Rate --------------- ------- --------- --------- ------- --------- --------- 500.perlbench_r 32 804 63.4 S 32 759 67.1 * 500.perlbench_r 32 803 63.4 S 32 759 67.1 S 500.perlbench_r 32 803 63.4 * 32 761 66.9 S 502.gcc_r 32 482 94.0 S 32 437 104 * 502.gcc_r 32 483 93.8 S 32 438 104 S 502.gcc_r 32 483 93.8 * 32 437 104 S 505.mcf_r 32 436 119 S 32 436 119 S 505.mcf_r 32 437 118 * 32 437 118 * 505.mcf_r 32 438 118 S 32 438 118 S 520.omnetpp_r 32 740 56.7 * 32 683 61.4 * 520.omnetpp_r 32 759 55.3 S 32 686 61.2 S 520.omnetpp_r 32 738 56.9 S 32 679 61.9 S 523.xalancbmk_r 32 367 92.2 * 32 321 105 S 523.xalancbmk_r 32 365 92.5 S 32 321 105 * 523.xalancbmk_r 32 368 91.9 S 32 389 86.9 S 525.x264_r 32 291 192 S 32 290 193 S 525.x264_r 32 292 192 * 32 289 194 * 525.x264_r 32 298 188 S 32 288 194 S 531.deepsjeng_r 32 443 82.9 * 32 443 82.9 * 531.deepsjeng_r 32 440 83.4 S 32 440 83.4 S 531.deepsjeng_r 32 456 80.4 S 32 456 80.4 S 541.leela_r 32 741 71.5 S 32 733 72.3 S 541.leela_r 32 741 71.5 S 32 734 72.2 S 541.leela_r 32 741 71.5 * 32 734 72.2 * 548.exchange2_r 32 404 208 S 32 404 208 S 548.exchange2_r 32 401 209 S 32 401 209 S 548.exchange2_r 32 401 209 * 32 401 209 * 557.xz_r 32 578 59.8 S 32 578 59.8 S 557.xz_r 32 579 59.7 S 32 579 59.7 S 557.xz_r 32 579 59.7 * 32 579 59.7 * ================================================================================= 500.perlbench_r 32 803 63.4 * 32 759 67.1 * 502.gcc_r 32 483 93.8 * 32 437 104 * 505.mcf_r 32 437 118 * 32 437 118 * 520.omnetpp_r 32 740 56.7 * 32 683 61.4 * 523.xalancbmk_r 32 367 92.2 * 32 321 105 * 525.x264_r 32 292 192 * 32 289 194 * 531.deepsjeng_r 32 443 82.9 * 32 443 82.9 * 541.leela_r 32 741 71.5 * 32 734 72.2 * 548.exchange2_r 32 401 209 * 32 401 209 * 557.xz_r 32 579 59.7 * 32 579 59.7 * SPECrate2017_int_base 93.7 SPECrate2017_int_peak 97.5 HARDWARE -------- CPU Name: AMD EPYC 7261 Max MHz.: 2900 Nominal: 2500 Enabled: 16 cores, 2 chips, 2 threads/core Orderable: 1, 2 chip(s) Cache L1: 64 KB I + 32 KB D on chip per core L2: 512 KB I+D on chip per core L3: 64 MB I+D on chip per chip Other: None Memory: 256 GB (16 x 16 GB 1Rx4 PC4-2666V-L) Storage: 2 x 600 GB 10K SAS, RAID 1 Other: None SOFTWARE -------- OS: SUSE Linux Enterprise Server 12 (x86_64) SP3 Kernel 4.4.162-94.72-default Compiler: C/C++: Version 1.3.0 of AOCC Fortran: Version 4.8.2 of GCC Parallel: No Firmware: HPE BIOS Version A40 01/25/2019 released Feb-2019 File System: xfs System State: Run level 3 (multi-user) Base Pointers: 64-bit Peak Pointers: 32/64-bit Other: jemalloc memory allocator library v5.1.0 Compiler Notes -------------- The AMD64 AOCC Compiler Suite is available at http://developer.amd.com/amd-aocc/ The AOCC Fortran Plugin version 1.3.0 was used to leverage AOCC optimizers with gfortran. It is available here: http://developer.amd.com/amd-aocc/ jemalloc: configured and built with GCC v4.8.2 in RHEL v7.2 under default conditions. https://github.com/jemalloc/jemalloc/releases/download/5.1.0/jemalloc-5.1.0.tar.bz2 jemalloc uses environment variable MALLOC_CONF with values narenas and lg_chunk: narenas: sets the maximum number of arenas to use for automatic multiplexing of threads and arenas. lg_chunk: set the virtual memory chunk size (log base 2). For example, lg_chunk:21 sets the default chunk size to 2^21 = 2MiB. Submit Notes ------------ The config file option 'submit' was used. 'numactl' was used to bind copies to the cores. See the configuration file for details. Operating System Notes ---------------------- 'ulimit -s unlimited' was used to set environment stack size 'ulimit -l 2097152' was used to set environment locked pages in memory limit runspec command invoked through numactl i.e.: numactl --interleave=all runspec Set dirty_ratio=8 to limit dirty cache to 8% of memory Set swappiness=1 to swap only if necessary Set zone_reclaim_mode=1 to free local node memory and avoid remote memory sync then drop_caches=3 to reset caches before invoking runcpu dirty_ratio, swappiness, zone_reclaim_mode and drop_caches were all set using privileged echo (e.g. echo 1 > /proc/sys/vm/swappiness). Transparent huge pages set to 'always' for this run (OS default) General Notes ------------- Environment variables set by runcpu before the start of the run: LD_LIBRARY_PATH = "/home/cpu2017_rate/amd1812na_rate_revA_lib/64;/home/cpu2017_rate/amd1812na_rate_revA_lib/32:" Binaries were compiled on a system with 2p AMD EPYC 7601 CPU + 512GB Memory using RHEL 7.6 NA: The test sponsor attests, as of date of publication, that CVE-2017-5754 (Meltdown) is mitigated in the system as tested and documented. Yes: The test sponsor attests, as of date of publication, that CVE-2017-5753 (Spectre variant 1) is mitigated in the system as tested and documented. Yes: The test sponsor attests, as of date of publication, that CVE-2017-5715 (Spectre variant 2) is mitigated in the system as tested and documented. jemalloc: configured and built with GCC v4.8.2 in RHEL v7.2 under default conditions. jemalloc: sources available from jemalloc.net or https://github.com/jemalloc/jemalloc/releases jemalloc uses environment variable MALLOC_CONF with values narenas and lg_chunk: narenas: sets the maximum number of arenas to use for automatic multiplexing of threads and arenas. lg_chunk: set the virtual memory chunk size (log base 2). For example, lg_chunk:21 sets the default chunk size to 2^21 = 2MiB. Platform Notes -------------- BIOS Configuration: Thermal Configuration set to Maximum Cooling Performance Determinism set to Power Deterministic Memory Patrol Scrubbing set to Disabled Workload Profile set to General Throughput Compute Sysinfo program /home/cpu2017_rate/bin/sysinfo Rev: r5974 of 2018-05-19 9bcde8f2999c33d61f64985e45859ea9 running on linux-ubi7 Tue Jun 11 02:59:29 2019 SUT (System Under Test) info as seen by some common utilities. For more information on this section, see https://www.spec.org/cpu2017/Docs/config.html#sysinfo From /proc/cpuinfo model name : AMD EPYC 7261 8-Core Processor 2 "physical id"s (chips) 32 "processors" cores, siblings (Caution: counting these is hw and system dependent. The following excerpts from /proc/cpuinfo might not be reliable. Use with caution.) cpu cores : 8 siblings : 16 physical 0: cores 0 4 8 12 16 20 24 28 physical 1: cores 0 4 8 12 16 20 24 28 From lscpu: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 2 NUMA node(s): 8 Vendor ID: AuthenticAMD CPU family: 23 Model: 1 Model name: AMD EPYC 7261 8-Core Processor Stepping: 2 CPU MHz: 2500.000 CPU max MHz: 2500.0000 CPU min MHz: 1200.0000 BogoMIPS: 4990.58 Virtualization: AMD-V L1d cache: 32K L1i cache: 64K L2 cache: 512K L3 cache: 8192K NUMA node0 CPU(s): 0,1,16,17 NUMA node1 CPU(s): 2,3,18,19 NUMA node2 CPU(s): 4,5,20,21 NUMA node3 CPU(s): 6,7,22,23 NUMA node4 CPU(s): 8,9,24,25 NUMA node5 CPU(s): 10,11,26,27 NUMA node6 CPU(s): 12,13,28,29 NUMA node7 CPU(s): 14,15,30,31 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 mwaitx arat cpb hw_pstate ssbd ibpb npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold vmmcall avic fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 clzero irperf amd_ibpb overflow_recov succor smca /proc/cpuinfo cache data cache size : 512 KB From numactl --hardware WARNING: a numactl 'node' might or might not correspond to a physical chip. available: 8 nodes (0-7) node 0 cpus: 0 1 16 17 node 0 size: 32000 MB node 0 free: 31902 MB node 1 cpus: 2 3 18 19 node 1 size: 32254 MB node 1 free: 32145 MB node 2 cpus: 4 5 20 21 node 2 size: 32254 MB node 2 free: 32147 MB node 3 cpus: 6 7 22 23 node 3 size: 32254 MB node 3 free: 32168 MB node 4 cpus: 8 9 24 25 node 4 size: 32254 MB node 4 free: 32168 MB node 5 cpus: 10 11 26 27 node 5 size: 32254 MB node 5 free: 32167 MB node 6 cpus: 12 13 28 29 node 6 size: 32254 MB node 6 free: 32168 MB node 7 cpus: 14 15 30 31 node 7 size: 32122 MB node 7 free: 32033 MB node distances: node 0 1 2 3 4 5 6 7 0: 10 16 16 16 32 32 32 32 1: 16 10 16 16 32 32 32 32 2: 16 16 10 16 32 32 32 32 3: 16 16 16 10 32 32 32 32 4: 32 32 32 32 10 16 16 16 5: 32 32 32 32 16 10 16 16 6: 32 32 32 32 16 16 10 16 7: 32 32 32 32 16 16 16 10 From /proc/meminfo MemTotal: 263835136 kB HugePages_Total: 0 Hugepagesize: 2048 kB /usr/bin/lsb_release -d SUSE Linux Enterprise Server 12 SP3 From /etc/*release* /etc/*version* SuSE-release: SUSE Linux Enterprise Server 12 (x86_64) VERSION = 12 PATCHLEVEL = 3 # This file is deprecated and will be removed in a future service pack or release. # Please check /etc/os-release for details about this release. os-release: NAME="SLES" VERSION="12-SP3" VERSION_ID="12.3" PRETTY_NAME="SUSE Linux Enterprise Server 12 SP3" ID="sles" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:suse:sles:12:sp3" uname -a: Linux linux-ubi7 4.4.162-94.72-default #1 SMP Mon Nov 12 18:57:45 UTC 2018 (9de753f) x86_64 x86_64 x86_64 GNU/Linux Kernel self-reported vulnerability status: CVE-2017-5754 (Meltdown): Not affected CVE-2017-5753 (Spectre variant 1): Mitigation: __user pointer sanitization CVE-2017-5715 (Spectre variant 2): Mitigation: Full AMD retpoline, IBPB, RSB filling run-level 3 Jun 11 02:52 SPEC is set to: /home/cpu2017_rate Filesystem Type Size Used Avail Use% Mounted on /dev/sda4 xfs 518G 48G 470G 10% /home Additional information from dmidecode follows. WARNING: Use caution when you interpret this section. The 'dmidecode' program reads system data which is "intended to allow hardware to be accurately determined", but the intent may not be met, as there are frequent changes to hardware, firmware, and the "DMTF SMBIOS" standard. BIOS HPE A40 01/25/2019 Memory: 16x HPE 840757-191 16 GB 1 rank 2666 16x UNKNOWN NOT AVAILABLE (End of data from sysinfo program) Compiler Version Notes ---------------------- ============================================================================== CC 502.gcc_r(peak) ------------------------------------------------------------------------------ AOCC.LLVM.1.3.0.B34.2018_10_22 clang version 7.0.0 (CLANG: Jenkins AOCC_1_3_0_Release-Build#34) (based on LLVM AOCC.LLVM.1.3.0.B34.2018_10_22) Target: i386-unknown-linux-gnu Thread model: posix InstalledDir: /root/work/compilers/aocc1.3.0/AOCC-1.3.0-Compiler/bin ------------------------------------------------------------------------------ ============================================================================== CXXC 523.xalancbmk_r(peak) ------------------------------------------------------------------------------ AOCC.LLVM.1.3.0.B34.2018_10_22 clang version 7.0.0 (CLANG: Jenkins AOCC_1_3_0_Release-Build#34) (based on LLVM AOCC.LLVM.1.3.0.B34.2018_10_22) Target: i386-unknown-linux-gnu Thread model: posix InstalledDir: /root/work/compilers/aocc1.3.0/AOCC-1.3.0-Compiler/bin ------------------------------------------------------------------------------ ============================================================================== CC 500.perlbench_r(base) 502.gcc_r(base) 505.mcf_r(base, peak) 525.x264_r(base) 557.xz_r(base, peak) ------------------------------------------------------------------------------ AOCC.LLVM.1.3.0.B34.2018_10_22 clang version 7.0.0 (CLANG: Jenkins AOCC_1_3_0_Release-Build#34) (based on LLVM AOCC.LLVM.1.3.0.B34.2018_10_22) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /root/work/compilers/aocc1.3.0/AOCC-1.3.0-Compiler/bin ------------------------------------------------------------------------------ ============================================================================== CXXC 520.omnetpp_r(base, peak) 523.xalancbmk_r(base) 531.deepsjeng_r(base, peak) 541.leela_r(base) ------------------------------------------------------------------------------ AOCC.LLVM.1.3.0.B34.2018_10_22 clang version 7.0.0 (CLANG: Jenkins AOCC_1_3_0_Release-Build#34) (based on LLVM AOCC.LLVM.1.3.0.B34.2018_10_22) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /root/work/compilers/aocc1.3.0/AOCC-1.3.0-Compiler/bin ------------------------------------------------------------------------------ ============================================================================== CC 500.perlbench_r(peak) 525.x264_r(peak) ------------------------------------------------------------------------------ AOCC.LLVM.1.3.0.B34.2018_10_22 clang version 7.0.0 (CLANG: Jenkins AOCC_1_3_0_Release-Build#34) (based on LLVM AOCC.LLVM.1.3.0.B34.2018_10_22) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /root/work/compilers/aocc1.3.0/AOCC-1.3.0-Compiler/bin ------------------------------------------------------------------------------ ============================================================================== CXXC 541.leela_r(peak) ------------------------------------------------------------------------------ AOCC.LLVM.1.3.0.B34.2018_10_22 clang version 7.0.0 (CLANG: Jenkins AOCC_1_3_0_Release-Build#34) (based on LLVM AOCC.LLVM.1.3.0.B34.2018_10_22) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /root/work/compilers/aocc1.3.0/AOCC-1.3.0-Compiler/bin ------------------------------------------------------------------------------ ============================================================================== FC 548.exchange2_r(base, peak) ------------------------------------------------------------------------------ GNU Fortran (GCC) 4.8.2 Copyright (C) 2013 Free Software Foundation, Inc. GNU Fortran comes with NO WARRANTY, to the extent permitted by law. You may redistribute copies of GNU Fortran under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING ------------------------------------------------------------------------------ Base Compiler Invocation ------------------------ C benchmarks: clang C++ benchmarks: clang++ Fortran benchmarks: clang gfortran Base Portability Flags ---------------------- 500.perlbench_r: -DSPEC_LINUX_X64 -DSPEC_LP64 502.gcc_r: -DSPEC_LP64 505.mcf_r: -DSPEC_LP64 520.omnetpp_r: -DSPEC_LP64 523.xalancbmk_r: -DSPEC_LINUX -DSPEC_LP64 525.x264_r: -DSPEC_LP64 531.deepsjeng_r: -DSPEC_LP64 541.leela_r: -DSPEC_LP64 548.exchange2_r: -DSPEC_LP64 557.xz_r: -DSPEC_LP64 Base Optimization Flags ----------------------- C benchmarks: -flto -Wl,-mllvm -Wl,-function-specialize -Wl,-mllvm -Wl,-enable-vectorize-compares -O3 -ffast-math -march=znver1 -mno-avx2 -fstruct-layout=3 -mllvm -unroll-threshold=50 -fremap-arrays -mllvm -inline-threshold=1000 -flv-function-specialization -mllvm -enable-gvn-hoist -mllvm -function-specialize -z muldefs -lpthread -ldl -ljemalloc -lamdlibm C++ benchmarks: -flto -Wl,-mllvm -Wl,-function-specialize -Wl,-mllvm -Wl,-enable-vectorize-compares -O3 -march=znver1 -mllvm -unroll-threshold=100 -finline-aggressive -fremap-arrays -mllvm -inline-threshold=1000 -mllvm -enable-vectorize-compares=false -z muldefs -lpthread -ldl -ljemalloc -lamdlibm Fortran benchmarks: -Wl,-mllvm -Wl,-inline-recursion=4 -Wl,-mllvm -Wl,-lsr-in-nested-loop -Wl,-mllvm -Wl,-enable-iv-split -Wl,-mllvm -Wl,-merge-constant -Wl,-mllvm -Wl,-unroll-aggressive -Wl,-mllvm -Wl,-unroll-threshold=150 -flto -Wl,-mllvm -Wl,-function-specialize -Wl,-mllvm -Wl,-enable-vectorize-compares -O3 -mavx -madx -funroll-loops -ffast-math -frepack-arrays -z muldefs -fplugin=dragonegg.so -specs=integrated-as.specs -fplugin-arg-dragonegg-llvm-option=-disable-indvar-simplify -fplugin-arg-dragonegg-llvm-option=-unroll-aggressive -fplugin-arg-dragonegg-llvm-option=-unroll-threshold:150 -lpthread -ldl -ljemalloc -lamdlibm -lgfortran Peak Compiler Invocation ------------------------ C benchmarks: clang C++ benchmarks: clang++ Fortran benchmarks: clang gfortran Peak Portability Flags ---------------------- 500.perlbench_r: -DSPEC_LINUX_X64 -DSPEC_LP64 502.gcc_r: -D_FILE_OFFSET_BITS=64 505.mcf_r: -DSPEC_LP64 520.omnetpp_r: -DSPEC_LP64 523.xalancbmk_r: -DSPEC_LINUX -D_FILE_OFFSET_BITS=64 525.x264_r: -DSPEC_LP64 531.deepsjeng_r: -DSPEC_LP64 541.leela_r: -DSPEC_LP64 548.exchange2_r: -DSPEC_LP64 557.xz_r: -DSPEC_LP64 Peak Optimization Flags ----------------------- C benchmarks: 500.perlbench_r: -flto -Wl,-mllvm -Wl,-function-specialize -Wl,-mllvm -Wl,-enable-vectorize-compares -fprofile-instr-generate(pass 1) -fprofile-instr-use(pass 2) -Ofast -march=znver1 -fstruct-layout=3 -mllvm -vectorize-memory-aggressively -mno-avx2 -mllvm -unroll-threshold=100 -fremap-arrays -mllvm -inline-threshold=1000 -lpthread -ldl -ljemalloc 502.gcc_r: -m32 -flto -Wl,-mllvm -Wl,-function-specialize -Wl,-mllvm -Wl,-enable-vectorize-compares -Ofast -march=znver1 -fstruct-layout=3 -mllvm -vectorize-memory-aggressively -mno-avx2 -mllvm -unroll-threshold=100 -fremap-arrays -mllvm -inline-threshold=1000 -fgnu89-inline -lpthread -ldl -ljemalloc 505.mcf_r: basepeak = yes 525.x264_r: -flto -Wl,-mllvm -Wl,-function-specialize -Wl,-mllvm -Wl,-enable-vectorize-compares -fprofile-instr-generate(pass 1) -fprofile-instr-use(pass 2) -Ofast -march=znver1 -mno-avx2 -fstruct-layout=5 -mllvm -vectorize-memory-aggressively -mllvm -unroll-threshold=50 -fremap-arrays -mllvm -inline-threshold=1000 -mllvm -enable-gvn-hoist -flv-function-specialization -lamdlibm -ljemalloc -lpthread -ldl 557.xz_r: basepeak = yes C++ benchmarks: 520.omnetpp_r: -flto -Wl,-mllvm -Wl,-function-specialize -Wl,-mllvm -Wl,-enable-vectorize-compares -Ofast -march=znver1 -finline-aggressive -mllvm -unroll-threshold=100 -fremap-arrays -mllvm -inline-threshold=1000 -lpthread -ldl -ljemalloc 523.xalancbmk_r: -m32 -flto -Wl,-mllvm -Wl,-function-specialize -Wl,-mllvm -Wl,-enable-vectorize-compares -Ofast -march=znver1 -finline-aggressive -mllvm -unroll-threshold=100 -fremap-arrays -mllvm -inline-threshold=1000 -lpthread -ldl -ljemalloc 531.deepsjeng_r: basepeak = yes 541.leela_r: -flto -Wl,-mllvm -Wl,-function-specialize -Wl,-mllvm -Wl,-enable-vectorize-compares -fprofile-instr-generate(pass 1) -fprofile-instr-use(pass 2) -Ofast -march=znver1 -mllvm -unroll-count=8 -mllvm -unroll-threshold=100 -lpthread -ldl -ljemalloc Fortran benchmarks: 548.exchange2_r: basepeak = yes Peak Other Flags ---------------- C benchmarks: 502.gcc_r: -L/root/work/lib/jemalloc510/lib32 C++ benchmarks: 523.xalancbmk_r: -L/root/work/lib/jemalloc510/lib32 The flags files that were used to format this result can be browsed at http://www.spec.org/cpu2017/flags/aocc130-flags-revA-I.html http://www.spec.org/cpu2017/flags/gcc.2017-11-20.html http://www.spec.org/cpu2017/flags/HPE-Platform-Flags-AMD-V1.2-EPYC-revD.html You can also download the XML flags sources by saving the following links: http://www.spec.org/cpu2017/flags/aocc130-flags-revA-I.xml http://www.spec.org/cpu2017/flags/gcc.2017-11-20.xml http://www.spec.org/cpu2017/flags/HPE-Platform-Flags-AMD-V1.2-EPYC-revD.xml SPEC is a registered trademark of the Standard Performance Evaluation Corporation. All other brand and product names appearing in this result are trademarks or registered trademarks of their respective holders. -------------------------------------------------------------------------------------------------------------------- For questions about this result, please contact the tester. For other inquiries, please contact info@spec.org. Copyright 2017-2019 Standard Performance Evaluation Corporation Tested with SPEC CPU2017 v1.0.5 on 2019-06-10 20:59:28-0400. Report generated on 2019-07-22 11:40:44 by CPU2017 ASCII formatter v6067. Originally published on 2019-07-21.