Christian Collberg
Skip Abstract Section
Abstract
To encourage repeatable research, fund repeatability engineering and reward commitments to sharing research artifacts.
- Collberg, C., Proebsting, T., and Warren, A.M. Repeatability and Benefaction in Computer Systems Research: A Study and a Modest Proposal. Technical Report TR 14-04. Department of Computer Science, University of Arizona, Tucson, AZ, Dec. 2014; http://repeatability.cs.arizona.edu/v2/RepeatabilityTR.pdfGoogle Scholar
- Feitelson, D.G. From repeatability to reproducibility and corroboration. SIGOPS Operating Systems Review 49, 1 (Jan. 2015), 3--11. Google ScholarDigital Library
- Friedman, B. and Schneider, F.B. Incentivizing Quality and Impact: Evaluating Scholarship in Hiring, Tenure, and Promotion. Computing Research Association Best Practices Memo, Feb. 2015; http://archive2.cra.org/uploads/documents/resources/bpmemos/BP_Memo.pdfGoogle Scholar
- Gorp, P.V. and Mazanek, S. Share: A Web portal for creating and sharing executable research papers. Procedia Computer Science 4 (2011), 589--597.Google ScholarCross Ref
- Kovac̆ević, J. How to encourage and publish reproducible research. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Volume IV (Honolulu, HI, Apr. 15--20). IEEE Computer Society, 2007, 1273--1276.Google ScholarCross Ref
- Li-Thiao-Té, S. Literate program execution for reproducible research and executable papers. Procedia Computer Science 9 (2012), 439--448.Google ScholarCross Ref
- National Science Foundation. Grant Policy Manual 05-131. Arlington, VA, July 2005; http://www.nsf.gov/pubs/manuals/gpm05_131Google Scholar
- Nosek, B.A. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science 7, 6 (Nov. 2012), 657--660.Google Scholar
- Perianayagam, S., Andrews, G.R., and Hartman, J.H. Rex: A toolset for reproducing software experiments. In Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine (Hong Kong, Dec. 18--21). IEEE Computer Society, 2010, 613--617.Google Scholar
- Rozier, K.Y. and Rozier, E.W.D. Reproducibility, correctness, and buildability: The three principles for ethical public dissemination of computer science and engineering research. In Proceedings of the IEEE International Symposium on Ethics in Science, Technology, and Engineering (Chicago, IL, May 23--24). IEEE Computer Society, 2014, 1--13.Google Scholar
- Sandve, G.K., Nekrutenko, A., Taylor, J., and Hovig, E. Ten simple rules for reproducible computational research. PLoS Computational Biology 9, 10 (Oct. 24, 2013).Google ScholarCross Ref
- Schwab, M., Karrenbach, N., and Claerbout, J. Making scientific computations reproducible. Computing in Science Engineering 2, 6 (Nov. 2000), 61--67. Google ScholarDigital Library
- Stodden, V. Enabling reproducible research: Licensing for scientific innovation. International Journal of Communications Law & Policy 13 (Winter 2009), 22--46.Google Scholar
- Stodden, V. The legal framework for reproducible scientific research: Licensing and copyright. IEEE Computing in Science and Engineering 11, 1 (Jan.-Feb. 2009), 35--40. Google ScholarDigital Library
- Stodden, V. The Scientific Method in Practice: Reproducibility in the Computational Sciences. Technical Report Working Paper 4773-10. MIT Sloan School of Management, Cambridge, MA, Feb. 2010; http://web.stanford.edu/~vcs/papers/SMPRCS2010.pdfGoogle ScholarCross Ref
- Stodden, V., Borwein, J., and Bailey, D.H. 'Setting the default to reproducible' in computational science research. SIAM News 46, 5 (June 2013).Google Scholar
- Stodden, V., Hurlin, C., and Perignon, C. RunMyCode.org: A novel dissemination and collaboration platform for executing published computational results. In Proceedings of the Eighth IEEE International Conference on E-Science (Chicago, IL, Sept. 15). IEEE Computer Society, 2012, 1--8. Google ScholarDigital Library
- Vandewalle, P., Kovac̆ević, J., and Vetterli, M. Reproducible research in signal processing: What, why, and how. IEEE Signal Processing Magazine 26, 3 (May 2009), 37--47.Google ScholarCross Ref
- Vitek, J. and Kalibera, T. Repeatability, reproducibility, and rigor in systems research. In Proceedings of the 11th ACM International Conference on Embedded Software (Taipei, Taiwan, Oct. 9--14). ACM Press, New York, 2011, 33--38. Google ScholarDigital Library
- Yale Law School Roundtable on Data and Code Sharing. Reproducible research: Addressing the need for data and code sharing in computational science. Computing in Science and Engineering 12, 5 (Sept./Oct. 2010), 8--13.Google Scholar
Index Terms
- Repeatability in computer systems research
Recommendations
From Repeatability to Reproducibility and Corroboration
Special Issue on Repeatability and Sharing of Experimental ArtifactsBeing able to repeat experiments is considered a hallmark of the scientific method, used to confirm or refute hypotheses and previously obtained results. But this can take many forms, from precise repetition using the original experimental artifacts, to ...
Towards repeatability & verifiability in networking experiments
Growing complexity of networking experiments has rendered the accuracy, repetition and reproduction of experimental results indispensable. This research work aims at developing a tool to facilitate repeatability and verifiability of experiments on ...
KheOps: Cost-effective Repeatability, Reproducibility, and Replicability of Edge-to-Cloud Experiments
ACM REP '23: Proceedings of the 2023 ACM Conference on Reproducibility and ReplicabilityDistributed infrastructures for computation and analytics are now evolving towards an interconnected ecosystem allowing complex scientific workflows to be executed across hybrid systems spanning from IoT Edge devices to Clouds, and sometimes to ...
Comments