Reproducible Research @ AGILE '22
The Reproducible AGILE team is happy to contribute to the AGILE conference 2022 by organising two activities:
- Reproducibility review
- Workshop Replication in geospatial research
For the third time, a reproducibility review is be conducted with all accepted full papers based on the mandatory reproducible paper guidelines.
The reproducibility reviews are published on OSF at 👉 https://osf.io/r5w79/ 👈.
Reproducible AGILE community member Alexander Kmoch presented an overview on the reproducibility review process and its results on behalf of reproducibility committee chair Daniel Nüst at the final day of the conference.
The slides are published at https://doi.org/10.5281/zenodo.6625206.
A more extensive written report is published in the document about the reproducibility review process in the weeks after the conference.
Find all details about the conference and the submission guidelines, including information about the reproducibility guidelines and review on the conference website: https://agile-online.org/conference-2022.
If you want to get a quick overview about previous reproducibility reviews, take a look at the presentation by reproducibility committee chair Daniel Nüst at the Munin conference.
All non-rejected submissions, i.e., papers that are accepted or have minor revisions only, will be screened for reproducibility based on the information provided in the Data and Software Availability section.
The reproducibility review happens after the scientific review, because some papers are irreproducible for good reason (e.g., conceptual work) and are still published.
Successfully reproduced papers will receive the Reproducible AGILE badge, see the 2020 proceedings for some examples.
Learn more about the process in the review process and reports document.
‼️ We are looking for more reproducibility reviewers ‼️
If you are interested to learn more about computational reproducibility and contribute to the cultural change in GIScience and in the AGILE community, please nominate yourself to become a reproducibility reviewer by sending an email to Daniel.
See below for more information.
- Daniel Nüst, Chair (University of Münster, Germany)
- Carlos Granell (Universitat of Jaume I, Spain)
- Eftychia Koukouraki (University of Münster, Germany)
- Eleni Tomai (Cartography Laboratory, National Technical University of Athens, Greece)
- Frank Ostermann (University of Twente, The Netherlands)
- Jakub Krukar (University of Münster, Germany)
- Philipp Friese (Technical University of Munich, Germany)
- Rémy Decoupes (INRAE, UMR TETIS, Université de Montpellier, France)
What is expected from reproducibility reviewers?
- An interest to learn more about computational reproducibility.
A good starting point are the AGILE Reproducible Paper Guidelines, which you can familiarise yourself with.
- Any skills with different software, tools, or programming languages are welcome, but not strictly necessary.
We expect authors to provide good enough instructions for anyone to reproduce their work, so previous skills are really just important to ensure a time effective reproducibility review. Note that you can always reach out to fellow reproducibility reviewers for advice and help, and most importantly, the authors. A reproduction can be collaborative endavour for author and reproducibility reviewer.
- Some time in April/May 2022 (see conference schedule). The reproducibility reviews will be conducted after notification of full paper acceptance and must be completed before camera-ready papers are due so that the reference to the reproducibility report can be added.
Based on previous years, a reproducibility review (executing the workflow & writing a report) takes between 2-4 hours, not counting the actual computations. This time is likely to be spread out across a couple of weeks as communications with authors take place and is comparable to the effort of a scientific review.
You can find a more elaborate description of the tasks in the Reproducibility Reviewer Guidelines.
How do I become a reproducibility reviewer?
- Register on the AGILE Discourse server.
- Send an email to Daniel with a few words about why you want to join and your user name for the AGILE Discourse server, your email for accessing non-public Google Docs, and a link to your profile on OSF.
- Daniel will send you a short welcome message and invite you to the internal reproducibility reviewer discussion forum, where you can post your interests and skills, so that you can be matched to a submission when the reproductions start.
This site (https://reproducible-agile.github.io) does not
collect or store any personal data nor uses it cookies.
Embedded Twitter widgets on this site do not collect
information used for personalization.
Although we check the content carefully, we cannot accept responsibility for the content of external links. The linked sites’ carriers are responsible for their sites’ content.