Daniel Citron
Abstract
A majority of the papers published in leading computer architecture conferences use SPEC CPU2000, or its predecessor SPEC CPU95, which has become the de facto standard for measuring processor and/or memory-hierarchy performance. However, in most cases a subset of the suite's benchmarks are simulated. For example: 27 papers were published in ISCA 2002, 16 used SPEC CINT2000, 4 used the whole suite, and only 3 papers explained their omissions.This paper quantifies the extent of this phenomenon in the ISCA, Micro, and HPCA conferences: 173 papers were surveyed, 115 used benchmarks from SPEC CINT, but only 23 used the whole suite. If this current trend continues, by the year 2005 80% of the papers will use the full CINT2000 suite, a year after CPU2004 shall be announced.We claim that results based upon a subset of a benchmark suite are speculative and conflict with Amdahl's Law. The law implies that we must present the speedup of using the proposed technique on the whole suite. Projecting the law (by statistically supplying values for the missing benchmarks) to several published papers reduces promising results to average ones. Speedups are reduced from 1.42 to 1.16 in one case, from 1.43 to 1.13 in another, and from 1.76 to 1.15 in a third.Finally, we have found that the disregard for CFP2000 is unwarranted in papers that explore the data cache domain, the suite displays a higher data cache miss rate than CINT2000, which is used more frequently.
- J. F. Cantin and M. D. Hill, "Cache Performance for SPEC CPU2000 Benchmarks", http://www.cs.wisc.edu/multifacet/misc/spec2000cachedata. Google Scholar
Digital Library
- D. Citron, D. Feitelson and L. Rudolph, "Accelerating Multi-Media Processing by Implementing Memoing in Multiplication and Division Units", Proc. of the 8th Int. Conf. on Architectural Support for Programming Languages and Operating Systems, pp. 252--261, October 1998. Google ScholarDigital Library
- R. Giladi and N. Ahituv, "SPEC as a Performance Evaluation Measure", Computer 28:8, pp. 33--44, August 1995. Google ScholarDigital Library
- J. L. Hennessy and D. A. Patterson, Computer Architecture: A Quantitative Approach, 3rd edition, Morgan Kaufmann Publishers, 2003. Google ScholarDigital Library
- AJ KleinOsowski and D. J. Lilja, "MinneSPEC: A New SPEC Benchmark Workload for Simulation-Based Computer Architecture Research", Computer Architecture Letters, June 2002. Google ScholarDigital Library
- T. Lafage and A. Seznec, "Choosing Representative Slices of Program Execution for Microarchitecture Simulations: A Preliminary Application to the Data Stream", in IEEE 3rd Annual Workshop on Workload Characterization (WWC-2000) held in Conjunction with the International Conference on Computer Design (ICCD-2000), September 2000.Google Scholar
- M. Moudgill, J. Wellman, and J. Moreno, "Environment for PowerPC Microarchitecture Exploration", IEEE Micro 19(3), pp. 15--25, May/June 1999. Google ScholarDigital Library
- S. Nussbaum and J. E. Smith, "Modeling Superscalar Processors via Statistical Simulation", in Proceedings of the 2001 International Conference on Parallel Architectures and Compilation Techniques (PACT-2001), pp. 15--24, September 2001. Google ScholarDigital Library
- D. Parikh, K. Skadron, Y. Zhang, M. Barcella, and M. R. Stan, "Power Issues Related to Branch Prediction", Proc. of the 8th Int. Symp. on High-Performance Computer Architecture, pp. 233--244, February 2002. Google ScholarDigital Library
- SimpleScalar LLC, http://www.simplescalar.com.Google Scholar
- Smith J. E., "Characterizing Computer Performance With a Single Number", Comm. ACM 31:10, pp. 1202--1206, October 1988. Google ScholarDigital Library
- Standard Performance Evaluation Corporation, CPU2000, http://www.spec.org/osg/cpu2000.Google Scholar
- Standard Performance Evaluation Corporation, CPU95, http://www.spec.org/osg/cpu95.Google Scholar
- SPEC CPU Benchmark Search Program, http://www.spec.org/cpu2000/CPU2004.Google Scholar
Recommendations
MisSPECulation: partial and misleading use of SPEC CPU2000 in computer architecture conferences
ISCA '03: Proceedings of the 30th annual international symposium on Computer architectureA majority of the papers published in leading computer architecture conferences use SPEC CPU2000, or its predecessor SPEC CPU95, which has become the de facto standard for measuring processor and/or memory-hierarchy performance. However, in most cases a ...
Reducing misspeculation overhead for module-level speculative execution
CF '05: Proceedings of the 2nd conference on Computing frontiersThread-level speculative execution is a technique that makes it possible for a wider range of single-threaded applications to make use of the processing resources in a chip multiprocessor.We consider module-level speculation, i.e., speculative threads ...
Reducing misspeculation penalty in trace-level speculative multithreaded architectures
ISHPC'05/ALPS'06: Proceedings of the 6th international symposium on high-performance computing and 1st international conference on Advanced low power systemsTrace-Level Speculative Multithreaded Processors exploit trace-level speculation by means of two threads working cooperatively. One thread, called the speculative thread, executes instructions ahead of the other by speculating on the result of several ...
Comments