Community Workshop on Practical Reproducibility in HPC

This workshop is co-located with a major Computer Science conference taking place in Atlanta, GA on November 18, 2024.

There is a broad agreement that in the digital age science should be shared digitally — through artifacts such as code and data — and that adopting practices enabling or facilitating reproducibility of computational results can lead to more robust science and increased scientific productivity. Many conferences encourage this practice through artifact evaluation initiatives — yet, the tools, services, and practices that support both authors and reviewers in sharing the research are still often either insufficient or lacking. This is particularly true in high performance computing (HPC) where packaging experiments for reproducibility presents unique challenges given their need for deep reconfiguration; unique and cutting edge hardware; and significant scales.

The objective of this workshop is to bring together experiment authors and reviewers with experience or insight into packaging and reviewing computer science experiments on open platforms to discuss challenges, share ideas and tips, and formulate a path forward on how to best support reproducibility for computer science research. Since many such initiatives were supported by the Chameleon platform (www.chameleoncloud.org), this workshop will also double as a Chameleon User Meeting with the objective to bring together both existing, potential, and future users of the platform.

As in previous Chameleon User Meetings, the organizers will reimburse travel expenses of up to $1,500 for the presenting authors of the top 10 selected presentation abstracts (one author per abstract). Please, take a look at the Call For Presentations below for details.

Important Dates and Actions

• Submission due date: Oct. 18, 2024
• Send submissions: Send submissions formatted as per the instructions below to: presentations@chameleoncloud.org
• Acceptance notification date: Oct. 21, 2024

See more details about the workshop on our event webpage About page.

Presentation Proposal Guidelines

Presentation proposals should be in PDF format, no longer than 2 pages in length, and should include the following information:

Preamble:

• Presentation title
• Presenter name(s) and affiliation(s)
• Reproducibility platforms/tools used or projected (e.g., Chameleon, other testbed, containers, notebooks)

Proposal Body Sections:

Project Description (one paragraph or less):

A brief but clear description of the Computer Science experiment you are packaging or reproducing: What research problem does it address? (What is the hypothesis? What is the challenge/trade-off? Why is it significant?) How does the experiment capture this research problem?

Reproducibility Approach (one paragraph or less):

What is the description of the experiment you are either packaging or reproducing? What resources does it need and how many? How do those resources need to be configured? What is the experimental environments in which the experiment should run? How are they configured? What does the experiment body consist of? How long and in what manner is the experiment run? What types of data does it produce and how much? How is the data analyzed? What platforms or tools did you use in your experiment?

Lessons Learned:

Good and bad: What were the most important obstacles to either packaging for reproducibility or reproducing the experiment you encountered – and how did you overcome them? Were they on the infrastructure level (hardware availability), configuration level (how the hardware is configured, i.e., “creating an MPI cluster”), executing the experimental workflow, managing or analyzing data? What strategies were effective? How did you decide (or what guidelines did you give to reviewers) as to when the experiment is reproduced? What suggestions or wishes do you have on how support for reproducibility should improve? Specifically, what should improve in (1) the ability to set up an experimental environment, (2) the ability to manage the experiment body, and (3) data analysis? Which of those desired capabilities are the most important ones? Any procedural insights on how to organize reproducibility initiatives?

Impact and Future Directions (one paragraph or less):

How has prioritizing reproducibility influenced your HPC research or applications? How can the HPC community better integrate reproducibility into mainstream research and education?

Selection Criteria

Presentations will be selected based on their relevance to HPC-specific reproducibility challenges, potential for fostering discussion, and the insights they offer to the community. We particularly encourage submissions that:

• Have an insightful, detailed, and well-communicated “lessons learned” section
• Have a good discussion/analysis of specific reproducibility challenges in HPC
• Share experiences from HPC-focused reproducibility initiatives

Travel Support

For the top 10 selected abstracts, we will reimburse travel expenses of up to $1,500 for the presenting authors (one per abstract). Submitting the presentation proposal doubles as travel support application.

Workshop Outcomes

The workshop aims to produce a report capturing the community’s collective knowledge and recommendations for advancing practical reproducibility in HPC. Your presentations and participation will directly contribute to this valuable resource for the HPC community.

Please send your submissions to presentations@chameleoncloud.org. For any questions, contact us at contact@chameleoncloud.org or via the Chameleon users list.

We look forward to your contributions and to seeing you in Atlanta this November!