Short Paper Postprints

Reproducible Research @ AGILE '21

Reproducibility review

The AGILE conference 2021 will be a fully virtual conference. For the second time, a reproducibility review will be conducted with tentatively accepted full papers. For the first time, the reproducibility guidelines will be mandatory.

Find all details about the conference and the submission guidelines, including information about the reproducibility guidelines and review on the conference website: If you want to get a quick overview about last year’s reproducibility review, take a look at the presentation by reproducibility committee chair Daniel Nüst at the Munin conference.

All non-rejected submissions, i.e., papers that are accepted or have minor revisions only, will be screen for reproducibility based on the information provided in the Data and Software Availability section. The reproducibility review happens after the scientific review, because irreproducible papers will still be published. Successfully reproduced papers will receive the Reproducible AGILE badge, see the 2020 proceedings for some examples.

Learn more about the process in the review process and reports document.

Reproducibility committee

‼️ We are looking for more reproducibility reviewers ‼️

If you are interested to learn more about computational reproducibility and contribute to the cultural change in GIScience and in the AGILE community, please nominate yourself to become a reproducibility reviewer by sending an email to Daniel. See below for more information.

What is expected from reproducibility reviewers?

  1. An interest to learn more about computational reproducibility. A good starting point are the AGILE Reproducible Paper Guidelines, which you can familiarise yourself with.
  2. Any skills with different software, tools, or programming languages are welcome, but not strictly necessary. We expect authors to provide good enough instructions for anyone to reproduce their work, so previous skills are really just important to ensure a time effective reproducibility review. Note that you can always reach out to fellow reproducibility reviewers for advice and help, and most importantly, the authors. A reproduction can be collaborative endavour for author and reproducibility reviewer.
  3. Some time in April 2021 (see conference schedule). The reproducibility reviews will be conducted after notification of full paper acceptance (April 5) and must be completed before camera-ready papers are due (May 3) so that the reference to the reproducibility report can be added. Based on previous years, a reproducibility review (executing the workflow & writing a report) takes between 2-4 hours, not counting the actual computations. This time is likely to be spread out across a couple of weeks as communications with authors take place and is comparable to the effort of a scientific review.

You can find a more elaborate description of the tasks in the Reproducibility Reviewer Guidelines.

How do I become a reproducibility reviewer?

  1. Register on the AGILE Discourse server.
  2. Send an email to Daniel with your Discourse user name and a few words about why you want to join.
  3. Daniel will send you a short welcome message and invite you to the internal reproducibility reviewer discussion forum, where you can post your interests and skills, so that you can be matched to a submission when the reproductions start.


Due to the obligations with the reproducibiltiy review, no workshops around the topic of reproducibility are currently planned. If you are interested to join the Reproducible AGILE team and contribute to the organisation of a workshop at AGILE Conference 2021. See last year’s planned workshops for possible topics.

Creative Commons Licence
Except where otherwise noted content is licensed under a Creative Commons Attribution 4.0 International License.
Privacy policy / Disclaimer This site ( does not collect or store any personal data nor uses it cookies. Embedded Twitter widgets on this site do not collect information used for personalization. Although we check the content carefully, we cannot accept responsibility for the content of external links. The linked sites’ carriers are responsible for their sites’ content.