Experiment Suggestions for Reproducibility Hackathons

Machine Learning

Experimental patterns to facilitate reproducibility:

Artifact

Description

Connect Google Colab to a server on Chameleon

Allows for running notebooks with Colab frontend (widely used in ML) on a Chameleon backend.  This gives you Chameleon’s bare metal access, storage, memory, and GPU capabilities, and also lets you specify the exact software environment (e.g. OS, Python version, CUDA version, Python package versions) for better reproducibility.

Experiments:

Examples of Replicated ML experiments on Chameleon

Artifact

Replicates result from…

Reproducing “Deep Neural Nets: 33 years ago and 33 years from now”

Karpathy, “Deep Neural Nets: 33 years ago and 33 years from now (Invited Post)”, ICLR Blog Track, 2022. https://iclr-blog-track.github.io/2022/03/26/lecun1989/ 

Exploring the Role of Grammar and Word Choice in Bias Toward African American English (AAE) in Hate Speech Classification

Camille Harris, Matan Halevy, Ayanna Howard, Amy Bruckman, and Diyi Yang. 2022. Exploring the Role of Grammar and Word Choice in Bias Toward African American English (AAE) in Hate Speech Classification. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 789–798. https://doi.org/10.1145/3531146.3533144

Tacotron2 qualitative claims

J. Shen et al., “Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions,” 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, 2018, pp. 4779-4783, doi: 10.1109/ICASSP.2018.8461368.

Re: On Warm Starting Neural Network Training

Jordan T. Ash and Ryan P. Adams. 2020. On warm-starting neural network training. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020).

Re: An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale

An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. Alexander Kolesnikov Alexey Dosovitskiy Dirk Weissenborn Georg Heigold Jakob Uszkoreit Lucas Beyer Matthias Minderer Mostafa Dehghani Neil Houlsby Sylvain Gelly Thomas Unterthiner Xiaohua Zhai. ICLR (2021)

Suggestions:

These are some recent ML papers that have code and/or pretrained models available, which make them good candidates for reproducibility packages.

  1. ConvNext (convolutional neural networks) – Citation: Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie. A ConvNet for the 2020s. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR ’22), 2022. [PDF][Github][Colab demo with pre-trained model]
  2. MaxViT (vision transformer) – Citation: Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. MaxViT: Multi-Axis Vision Transformer. Proceedings of the 17th European Conference on Computer Vision (ECCV ’22), 2022.
    [    PDF][Github][Blog post]
  3. Diffusion models – Citation: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR ’22), 2022.  [PDF][Github]
  4. Whisper (speech) – Citation: Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, Ilya Sutskever. Robust Speech Recognition via Large-Scale Weak Supervision. Proceedings of the 40th International Conference on Machine Learning (PMLR ’22), 2022. [PDF][Github][Unofficial Colab]
  5. Tabular data classification with neural networks – Citation: Noah Hollmann, Samuel Müller, Katharina Eggensperger, Frank Hutter. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second.  In Proceedings of the 2022 Conference on Neural Information Processing Systems (NeurIPS ’22). [PDF][Github (includes Colab notebooks)]
  6. Image Denoising, Deblurring, Deraining, Dehazing and Enhancement with MLP – Citation: Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. MAXIM: Multi-Axis MLP for Image Processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR ’22). [PDF][Blog post][Github (includes notebooks)]
  7. Citation recommendation – Citation:  Nianlong Gu, Yingqiang Gao, Richard H.R. Hahnloser. Local Citation Recommendation with Hierarchical-Attention Text Encoder and SciBERT-based Reranking. In Proceedings of the 2022 European Conference on Information Retrieval (ECIR ’22) [PDF][Colab][Github]
  8. Zero-shot visual question answering – Citation: Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, Steven C.H. Hoi. Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP ’22 Findings). [PDF][Colab][YouTube]
  9. Scene graph generation – Citation: Yuren Cong, Michael Ying Yang, Bodo Rosenhahn. RelTR: Relation Transformer for Scene Graph Generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.  [PDF][Github (official)][Colab]
  10. 3D image generation from text description – Citation: Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, Ben Poole. Zero-Shot Text-Guided Object Generation with Dream Fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. [PDF] [Github] [Colab]
  11. Real-time background matting – Citation: Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan, Rynson W.H. Lau. MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition. Proceedings of the 2022 AAAI Conference on Artificial Intelligence (AAAI ’22). [PDF][Github (with links to Colab notebooks)][Video]
  12. Photo aging – Citation: Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, Ira Kemelmacher-Shlizerman.  Lifespan Age Transformation Synthesis. Proceedings of the 15th European Conference on Computer Vision (ECCV ’20), 2020. [PDF][Github][Colab]
  13. Music generation – Citation: Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, Alexandre Défossez. MusicGen: Simple and Controllable Music Generation. arXiv:2306.05284, 2023. [PDF][Github][Colab][Website][New Colab]
  14. Music transcription – Citation: Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, Jesse Engel. MT3: Multi-Task Multitrack Music Transcription. Proceedings of the 10th International Conference on Learning Representations (ICLR ’22), 2022. [PDF][Github][Colab]
  15. Learning periodic data – Citation: Yuzhe Yang, Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming-Zher Poh, Daniel McDuff. SimPer: Simple Self-Supervised Learning of Periodic Targets. Proceedings of the 11th International Conference on Learning Representations (ICLR ’23), 2023. [PDF][Github][Colab][Blog post]
  16. Protein folding – Citation: John Jumper, et al. Highly accurate protein structure prediction with AlphaFold. Nature 2021. [Paper] [Github (official)] [Colab]
  17. Connecting text and images – CLIP – Citation: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning (PMLR), 2021. [PDF][Github][Colab]
  18. Image captioning with CLIP – Citation: Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, Mohit Bansal. Fine-grained Image Captioning with CLIP Reward. Findings of the Association for Computational Linguistics (NAACL), 2022. [PDF][Github][Colab]
  19. Frame interpolation – Citation: Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, Brian Curless. FILM: Frame Interpolation for Large Motion. Proceedings of the 17th European Conference on Computer Vision (ECCV ’22), 2022. [Github][PDF][Colab]
  20. Text to video retrieval – Citation: Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, Andrew Zisserman. End-to-End Learning of Visual Representations from Uncurated Instructional Videos. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR ’20), 2020. [PDF][Website][Colab]
  21. Monocular depth estimation – Citation: Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, Matthias Müller. ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth. arXiv:2302.12288. 2023. [PDF][Github (with Colab)]
  22. Lip-syncing video to audio – Citation: K R Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, C V Jawahar. A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild. Proceedings of the 28th ACM International Conference on Multimedia (MM ’20). 2020. [PDF][Github with Colab]
  23. Talking face generation – Citation: Suzhen Wang, Lincheng Li, Yu Ding, Xin Yu. One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning. Proceedings of the 2022 AAAI Conference on Artificial Intelligence (AAAI ’22). 2022.
    [PDF][Github][Colab (unofficial)]
  24. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis- Citation: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2021. NeRF: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 1 (January 2022), 99–106. https://doi.org/10.1145/3503250  [PDF][Github (with Colab)] [Website]
  25. YOLO (fast object detection) – Citation: Chien-Yao Wang, Alexey Bochkovskiy, Hong-Yuan Mark Liao. YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 7464-7475. [PDF][Github (with Colab)]

Networking

Experimental patterns to facilitate reproducibility:

Artifact

Description

Network emulation

Networking experiments often require very specific, configurable, network conditions. This artifact shows how to emulate different network conditions (packet loss, delay, jitter, data rate) over a link in a Chameleon experiment.

Experiments:

Examples of replicated networking experiments on FABRIC. (Note: Chameleon has limited hardware resources with >2 NICs and none with >3 NICs, so some networking experiments are better suited for associated platforms, like FABRIC.)

Artifact

Replicates results from…

Replication: “When to Use and When Not to Use BBR”

Yi Cao, Arpit Jain, Kriti Sharma, Aruna Balasubramanian, and Anshul Gandhi. 2019. When to use and when not to use BBR: An empirical analysis and evaluation study. In Proceedings of the Internet Measurement Conference (IMC ’19). Association for Computing Machinery, New York, NY, USA, 130–136. https://doi.org/10.1145/3355369.3355579 

Some of the Internet may be heading towards BBR dominance: an experimental study

Ayush Mishra, Wee Han Tiu, and Ben Leong. 2022. Are we heading towards a BBR-dominant internet? In Proceedings of the 22nd ACM Internet Measurement Conference (IMC ’22). Association for Computing Machinery, New York, NY, USA, 538–550. https://doi.org/10.1145/3517745.3561429 

Examples of replicated networking experiments on other platforms

These could be good candidates for “packaging” for Chameleon and/or FABRIC.

Artifact

Replicates results from…

MPCC : Online learning multipath congestion control

Tomer Gilad, Neta Rozen-Schiff, P. Brighten Godfrey, Costin Raiciu, and Michael Schapira. 2020. MPCC: online learning multipath transport. In Proceedings of the 16th International Conference on emerging Networking EXperiments and Technologies (CoNEXT ’20). Association for Computing Machinery, New York, NY, USA, 121–135. https://doi.org/10.1145/3386367.3433030 

Do Switches Still Need to Deliver Packets in Sequence?

U. Usubütün, F. Fund and S. Panwar, “Do Switches Still Need to Deliver Packets in Sequence?,” 2023 IEEE 24th International Conference on High Performance Switching and Routing (HPSR), Albuquerque, NM, USA, 2023, pp. 89-95, doi: 10.1109/HPSR57248.2023.10147992

Suggestions:

These are some recent networking papers that are also good candidates for reproducibility packages.

  1. QueuePilot: Reviving Small Buffers With a Learned AQM Policy. Paper: M. Dery, O. Krupnik and I. Keslassy, “QueuePilot: Reviving Small Buffers With a Learned AQM Policy,” IEEE INFOCOM 2023 – IEEE Conference on Computer Communications, New York City, NY, USA, 2023, pp. 1-10, doi: 10.1109/INFOCOM53939.2023.10228975.
    Source code: https://github.com/2dm/QueuePilot/tree/main
  2. QUIC(k) Enough in the Long Run? Sustained Throughput Performance of QUIC Implementations. Paper: M. König, O. P. Waldhorst and M. Zitterbart, “QUIC(k) Enough in the Long Run? Sustained Throughput Performance of QUIC Implementations,” 2023 IEEE 48th Conference on Local Computer Networks (LCN), Daytona Beach, FL, USA, 2023, pp. 1-4, doi: 10.1109/LCN58197.2023.10223395.
  3. Understanding speciation in QUIC congestion control. Paper: Ayush Mishra, Sherman Lim, and Ben Leong. 2022. Understanding speciation in QUIC congestion control. In Proceedings of the 22nd ACM Internet Measurement Conference (IMC ’22). Association for Computing Machinery, New York, NY, USA, 560–566. https://doi.org/10.1145/3517745.3561459. Source code: https://github.com/NUS-SNL/QUICbench 
  4. Elephants Sharing the Highway: Studying TCP Fairness in Large Transfers over High Throughput Links. Paper: Imtiaz Mahmud, George Papadimitriou, Cong Wang, Mariam Kiran, Anirban Mandal, and Ewa Deelman. 2023. Elephants Sharing the Highway: Studying TCP Fairness in Large Transfers over High Throughput Links. In Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis (SC-W ’23). Association for Computing Machinery, New York, NY, USA, 806–818. https://doi.org/10.1145/3624062.3624594. Code and materials: https://github.com/PoSeiDon-Workflows/tcp-conflict-study.git 
  5. Multipath TCP Scheduling in the Age of Buffer Bloat Optimizations. Paper: D. Weber, C. Fuchs, N. Auler, B. Schütz and N. Aschenbruck, “Multipath TCP Scheduling in the Age of Buffer Bloat Optimizations,” 2023 IEEE 48th Conference on Local Computer Networks (LCN), Daytona Beach, FL, USA, 2023, pp. 1-8, doi: 10.1109/LCN58197.2023.10223336.
  6. A Principled Look at the Utility of Feedback in Congestion Control. Paper: Mohit P. Tahiliani, Vishal Misra, and K. K. Ramakrishnan. 2019. A Principled Look at the Utility of Feedback in Congestion Control. In Proceedings of the 2019 Workshop on Buffer Sizing (BS ’19). Association for Computing Machinery, New York, NY, USA, Article 8, 1–5. DOI:https://doi.org/10.1145/3375235.3375243

Data Management/Storage

Experimental patterns to facilitate reproducibility:

Artifact Description
ATC/OSDI 23 Tutorial Simple FIO Benchmark This is a simple FIO benchmark only on RAM disk and SSD. No special storage hardware is needed.
ATC/OSDI 23 Simple Filesystem Benchmark We will observe default filesystem performance (ext4) by simulating multiple FileBench workloads.
ATC/OSDI 23 Tutorial FIO Benchmark This is a sample artifact for running FIO benchmark on a single SSD, two SSD’s RAID0 group, and RAM disk. This is intended for the tutorial during ATC/OSDI23 BoF.

Experiments:

Artifact Replicates results from…
ATC/OSDI 2023 BoF SpecREDs Li, N., Kalaba, A., Freedman, M., Lloyd, W., & Levy, A. (2022). Speculative Recovery: Cheap, Highly Available Fault Tolerance with Disaggregated Storage. In 2022 USENIX Annual Technical Conference (USENIX ATC 22) (pp. 271–286). https://www.usenix.org/system/files/atc22-li-nanqinqin.pdf
ATC/OSDI 2023 BoF Roller Hongyu Zhu, Ruofan Wu, Yijia Diao, Shanbin Ke, Haoyu Li, Chen Zhang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Wei Cui, Fan Yang, Mao Yang, Lidong Zhou, Asaf Cidon and Gennady Pekhimenko (2022). ROLLER: Fast and Efficient Tensor Compilation for Deep Learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) (pp. 233-248).
ATC/OSDI 2023 BoF FlashNet In the ML/Deep Learning community, the large ImageNet benchmarks have spurred research in image recognition. Similarly, we would like to provide benchmarks for fostering storage research in ML-based per-IO latency prediction. Therefore, we present FlashNet, a reproducible data science platform for storage systems. To start a big task, we use I/O latency prediction as a case study. Thus, FlashNet has been built for I/O latency prediction tasks. With FlashNet, data engineers can collect the IO traces of various devices. The data scientists then can train the ML models to predict the IO latency based on those traces. All traces, results, and codes will be shared in the FlashNet training ground platform which utilizes Chameleon trovi for better reproducibility.
ATC/OSDI 2023 BOF Filesystems Benchmark (FileBench) We will observe three filesystem performance (ext4, XFS and btrfs) by simulating file server workloads inside Chameleon Storage Node SSDs.
ATC/OSDI 2023 BOF Local RDMA InfiniBand Performance Test Local RDMA InfiniBand performance test such as bandwidth and latency

Security

Experimental patterns to facilitate reproducibility: WIP

Experiments: WIP

Edge Computing

Experimental patterns to facilitate reproducibility: WIP

Experiments: WIP

Power Management

Experimental patterns to facilitate reproducibility: WIP

Experiments: WIP

An automated and portable method for selecting an optimal GPU frequency