Reproducible Research @ AGILE '24

The Reproducible AGILE team is happy to contribute to the AGILE conference 2024 by organising the reproducibility review, continueing an exemplary practice in Open Scholarship from the previous years. For the fifth time, a reproducibility review is conducted with all accepted full papers based on the mandatory reproducible paper guidelines.

Review details

Find all details about the conference and the submission guidelines, including information about the reproducibility guidelines and review on the conference website: https://agile-online.org/conference-2024.

If you want to get a quick overview about previous reproducibility reviews, take a look at the presentation by reproducibility committee chair Daniel Nüst at the Munin conference. In short, all non-rejected full paper submissions, i.e., papers that are accepted or have minor revisions only, will be screened for reproducibility based on the information provided in the Data and Software Availability section.

The reproducibility review happens after the scientific review, because some papers are irreproducible for good reason (e.g., conceptual work) and are still valuable scientific contributions. Successfully reproduced papers will receive the Reproducible AGILE badge, see the 2020 proceedings for some examples.

Learn more about the process in the review process and reports document.

Reproducibility committee

‼️ We are looking for more reproducibility reviewers ‼️

If you are interested to learn more about computational reproducibility and contribute to the cultural change in GIScience and in the AGILE community, please nominate yourself to become a reproducibility reviewer by sending an email to Carlos. See below for more information.

• Carlos Granell, Co-Chair (Universitat of Jaume I, Spain), carlos.granell@uji.es
• NN, Co-Chair
• …

What is expected from reproducibility reviewers?

1. An interest to learn more about computational reproducibility. A good starting point are the AGILE Reproducible Paper Guidelines, which you can familiarise yourself with.
2. Any skills with different software, tools, or programming languages are welcome, but not strictly necessary. We expect authors to provide good enough instructions for anyone to reproduce their work, so previous skills are really just important to ensure a time effective reproducibility review. Note that you can always reach out to fellow reproducibility reviewers for advice and help, and most importantly, the authors. A reproduction can be collaborative endavour for author and reproducibility reviewer.
3. Some time in April/May 2022 (see conference schedule). The reproducibility reviews will be conducted after notification of full paper acceptance and must be completed before camera-ready papers are due so that the reference to the reproducibility report can be added. Based on previous years, a reproducibility review (executing the workflow & writing a report) takes between 2-4 hours, not counting the actual computations. This time is likely to be spread out across a couple of weeks as communications with authors take place and is comparable to the effort of a scientific review.

You can find a more elaborate description of the tasks in the Reproducibility Reviewer Guidelines.

How do I become a reproducibility reviewer?

1. Register on the AGILE Discourse server.
2. Send an email to Carlos with a few words about why you want to join and your user name for the AGILE Discourse server, your email for accessing non-public Google Docs, and a link to your profile on OSF.
3. Carlos will send you a short welcome message and invite you to the internal reproducibility review coordination forum.