We’re excited to announce the publication of the Report on Challenges of Practical Reproducibility for Systems and HPC Computer Science, a culmination of our Community Workshop on Practical Reproducibility in HPC, held in November 2024 in Atlanta, GA (reproduciblehpc.org).
View or contribute to the experiment packaging and style checklists (appendix A and B) on our GitHub repository here.
Download the report here.
This comprehensive report emerged from the Community Workshop on Practical Reproducibility in HPC held on November 18, 2024, in Atlanta, Georgia, alongside SC24. The workshop brought together reproducibility practitioners, experiment authors, reviewers, and chairs of reproducibility initiatives to share experiences and formulate a path forward for community adoption of reproducibility practices.
The report details key challenges in practical reproducibility for systems and HPC research, including:
- Technical difficulties in packaging experiments and reviewing them
- Finding specialized hardware for reproduction
- Establishing experimental environments
- Defining reproducibility conditions
Most importantly, the report provides actionable recommendations for authors, reviewers, organizations, and the broader community, including detailed checklists for experiment packaging and style guidelines that can immediately improve artifact quality.
The workshop and report were supported by the NSF-sponsored REPETO project (award 2226406) and indirectly by the Chameleon project (NSF award 2027170).
Kate Keahey presenting her keynote at the Community Workshop on Practical Reproducibility in HPC (Nov. 2024 | Atlanta, GA).
