Reproducible Research @ AGILE '21

Reproducibility review

The AGILE conference 2021 is be a fully virtual conference. For the second time, a reproducibility review is be conducted with all (tentatively) accepted full papers. For the first time, the reproducibility guidelines are mandatory.

The reproducibility reviews are published on OSF at https://osf.io/h64sd/.

Report

Reproducibility committee chair Daniel Nüst reported on the progress of the initiative and the results of the 2021 reproducibility review in a decidated conference session on Friday, June 11th, 2021. The slides are available at https://doi.org/10.5281/zenodo.4926269.

9 papers of 15 accepted manuscripts were successfully (partially) reproduced. This corresponds to 60% percent of accepted papers.

Review details

Find all details about the conference and the submission guidelines, including information about the reproducibility guidelines and review on the conference website: https://agile-online.org/conference-2021.

If you want to get a quick overview about last year’s reproducibility review, take a look at the presentation by reproducibility committee chair Daniel Nüst at the Munin conference. All non-rejected submissions, i.e., papers that are accepted or have minor revisions only, will be screened for reproducibility based on the information provided in the Data and Software Availability section. The reproducibility review happens after the scientific review, because irreproducible papers will still be published. Successfully reproduced papers will receive the Reproducible AGILE badge, see the 2020 proceedings for some examples.

Learn more about the process in the review process and reports document.

Reproducibility committee

• Daniel Nüst, Chair (University of Münster, Germany)
• Frank Ostermann (University of Twente, The Netherlands)
• Carlos Granell (Universitat of Jaume I, Spain)
• Alexander Kmoch (University of Tartu, Estonia)
• Philipp Friese (Technical University of Munich, Germany)
• Anita Graser ( Austrian Institute of Technology, Austria)
• Jakub Krukar (University of Münster, Germany)

A description of the reproducibility reviewers’ tasks is in the Reproducibility Reviewer Guidelines.