SPEC CPU®2017 Credits

One of the most frequent questions SPEC® has been asked over the past decade has been "When is the next SPEC CPU® suite coming out?" This has not been easy to answer, in part because the SPEC CPU 2006 benchmark suite has been widely used.

SPEC CPU 2006 improved over its predecessors, and was intended to stand the test of time, as the computing industry continues to innovate. It has had a long and influential run.

A lot of work went into SPEC CPU 2006, and it paid off: as of June 2017, there are over 44,000 SPEC CPU 2006 results published at www.spec.org/cpu2006/results, for an average of ~338 submissions a month, ~78 a week or ~11 every day. And this doesn't include information on the runs that were not submitted to SPEC. Quite impressive for a benchmark that can take a bit of time and effort to set up and run.

Although there is a large established base set of results, the time has come to update things.

As with the previous SPEC CPU suites, the goal is to establish a suite of source code benchmarks to provide consistent, comparable measures of CPU-intensive performance. The task is not simple.

Challenges include:

• Locating meaningful programs.
• Porting them to numerous environments.
• Defining rules to govern how benchmarks are compiled, run and reported.
• Developing tools to help enforce these rules while still making things easy to use.
• Defining processes by which decisions can be made.

With SPEC CPU 2017, the SPEC CPU subcommittee believes that it has come up with a benchmark suite that builds upon and improves the previous suite, to meet the current and next generations of computing.

The new suite:

• Increases the number of applications and application areas.
• Uses more complex application source code.
• Uses more memory for the SPECspeed metrics.
• Allows parallelization for SPECspeed programs by using OpenMP directives.
• Improves the tools for building, running, and reporting.
• Adds an optional power metric.
• Clarifies and updates the run rules.
• Improves the documentation.

The SPEC CPU subcommittee hopes that the updates will continue to to be useful for people designing and comparing computing systems.

As with previous suites, there have been numerous people whose contributions have affected the end results, especially given the long development time.

SPEC CPU benchmarks are based primarily on real-world applications (rather than using loop kernels or artificial stress tests). We therefore -- most of all -- thank the authors of those real-world applications.

Thank you to contributors via the SPEC CPU Benchmark Search Program, contributors who work for member companies, and to the many authors of open source software. They are listed in SPEC CPU 2017 Licenses and in the documentation for individual benchmarks.

We would also like to thank:

• SPEC management, who provided us with resources, and kept the SPEC CPU Subcommittee on their toes:

  • Walter Bays - President, SPEC
  • Michael Paton, Alan Adamson, Steve Realmuto - Chairs of the SPEC Open Systems Steering Committee.

• The SPEC office, who were the ones who really provided the CPU subcommittee with the resources needed to get things done.

  • Dianne Rice, Kathy Powers, Diana Cercy, Charles McKay, Cathy Sandifer, Jason Glick

• SPEC supporting contributor managing the editorial cycle for SPEC CPU 2006 result review:

  • Julie Reilly

These are the people who spent time finding and porting code, performed testing and analysis, participating in the debate and discourse on the processes, rules and content of the suite. Much appreciation is given for their willingness to spend their talents and time working on SPEC CPU 2017 efforts and day to day SPEC work. Contributions included project leading benchmarks, providing regular resting reports, writing documentation or overseeing some aspect of the process. A little bit of each of these people can be found in the suite:

James Bucek Michael Carroll Mathew Colgrove Diego Esteves Darryl Gove John Henning

Hidekatsu Imura Hans Jøraandstad Willi Kratzer Subhash Kunnoth Alan MacKay Rahul Rahatekar

Jeff Reilly David Reiner Van Smith Cloyce Spradling Wilfried Stehling

Of particular note:

• Mat Colgrove led the largest number of benchmarks.
• Michael Carroll led the single largest benchmark.
• John Henning is the primary author of the documentation.
• Cloyce Spradling is the primary author of the SPEC CPU toolset.

Thanks are due to those who provided specific technical assistance, for example helping with analysis, testing, or addressing specific implementation issues:

Nelson Amaral Christopher Cambly Aaron Cragin Christopher Henning Jim Himer Jee Ho Lizy John Ajay Joshi

Arthur Kang Venkatesh KR Prathiba Kumar Carol LePage Manmohan Manoharan Chris Parrott Michael Paton Dave Raddatz

Srini Ramani David Schmidt Marcus Shawcroft Biswa Singh Suresh Srinivasan Raghavendra Swamy Jaya Yeruva Linda Zhang

And in memory of those whose past contributions, support and inspiration remain with us:

• Alan Adamson
• Kaivalya Dixit
• Larry Gray
• Tom Skornia

Finally, we thank all of the people behind the scenes in the compiler groups, architecture groups, and performance groups who supported their work for SPEC at their respective companies.