Workshop 'Replication in geospatial research' @ AGILE '22

The Reproducible AGILE team is happy to contribute to the AGILE conference 2022 by organising two activities:

1. Reproducibility review
2. Workshop Replication in geospatial research

1st AGILE Workshop on Replication in Geospatial Research

Update 08-06-22

Since we have not received sufficient submissions of replications, we decided to change the agenda as follows:

09:30 Round of introductions by everyone

09:45 Setting the scene: Replication in GIScience (presentation by workshop organizers)

10:15 Plenary discussion to arrive at a shared understanding of replication in GIScience

11:00 Break

11:15 Developing a concrete work plan for a joint paper to explore replication in GIScience (working in small teams)

12:45 Write-up and wrap-up

13:00 End of workshop

The objectives of the workshop have not changed (see below for more details). If you have questions, don’t hesitate to contact us!

About

Prior workshops at AGILE have investigated the need for reproducibility and its state at AGILE, leading to the establishing of a reproducibility review for all accepted full papers running for the third time. The logical next step is to investigate the related and important concept of replication. The discussion on replications and replicability in the GIScience community has just begun, so AGILE can again take a leading role and you can contribute to it by attending this workshop.

Replicating prior research based on a newly collected dataset or newly implemented analysis with different methods is important for verifying and advancing robust scientific knowledge. However, replicating research is currently not encouraged or rewarded since it rarely matches a journal’s scope or a funder’s call for projects. Further, there are epistemological and methodological challenges to replicability in geospatial research, such as the spatio-temporal non-stationarity of quantitative data and the context and subjectivity in qualitative research.

This half-day workshop aims to address and discuss these issues by asking participants to attempt a replication of a published study of their choice prior to the workshop and to report on this replication attempt by using a short report template. This template asks for feedback on challenges and an interpretation of the results. These replication studies are then presented at the workshop, followed by plenary and group discussions on

(a) the current role of replication studies in geospatial research,

(b) obstacles to replication and ways to overcome or work around them,

(c) what constitutes success or failure in a geospatial replication study.

After the workshop, at the minimum the reports and outcomes of the discussion are going to be integrated into a full journal paper with all contributors as co-authors. For inclusion as co-author, all code and data (license permitting) will have to be made available in an open repository, such as the Open Science Framework. Details will be decided as part of the workshop. Depending on interest by selected journals and the authors, another option is a special issue where the replication reports are expanded into full articles.

The workshop objectives are thus:

• To begin a systematic discussion on the role of replication studies in geospatial research
• To collect evidence on how to conduct a replication study and evaluate it
• To exchange experiences in replicating geospatial research
• To publish all submitted replication attempts in a joint replication study paper

Agenda

• 30 minutes introduction / setting the scene
• 60 minutes presentation (depending on number of submitted papers, these can be short or detailed)
• Break
• 90 minutes discussion in at least two groups and write-up of concrete outcomes
• Wrap up

Organisers

• Frank Ostermann (f.o.ostermann@utwente.nl)
• Markus Konkol (m.konkol@utwente.nl)
• Carlos Granell (carlos.granell@uji.es)

Call for submissions (CfP)

We invite all interested researchers to attempt a replication of a study of their choice, and to submit this work using this template. We encourage the use of an open repository to publish code and data already for the workshop contribution, as it will be mandatory for the planned journal article (see above). In addition to any other documentation, a repository should contain as official submission to the workshop a text document using the same structure as the template.

Some guidelines for the replication attempt:

• The study to be replicated should concern some sort of quantitative spatial analysis. While the replication of other forms of research is of no less importance, replicating a study involving participants and more qualitative analysis might be beyond the scope of this workshop contribution. However, the analysis to be replicated does not have to be the main focus of the published original study.
• There should be a scientific interest in replicating the original study. Of particular interest are any studies for which generalizability or transferability of outcomes would advance our body of knowledge, and for which no replication has yet been attempted. Generalizability and transferability apply to geographic and temporal scope (thus, replicating older studies would certainly be of interest) as well as robustness of analysis with respect to different methods.
• The replication should attempt a major change in either input data or methods used. In the former case, the replication tries to answer whether the observed effect in the original study is potentially generalizable to different geographic regions, different time periods, or even different phenomena (all depending on the way the new input data differs from the original). In the latter case, the interest in replication is whether the observed effect in the original study is also observable with different methods, or whether certain methods are more efficient and/or effective.
• Using a different software implementation of the same method (e.g., using R instead of Python or vice versa) is not considered a replication.
• The chosen study can be from the same author (team) as the replication study! It is perfectly fine to try and replicate one’s own work.
• All submitted reports will be subject to a short review by the workshop organizers to check whether the above-mentioned criteria (quantitative computation, spatial analysis, scientific interest in replication, sufficient information and evaluation) have been met.

If you are in doubt whether your plans are within scope and interest of the workshop, please do not hesitate to contact the organizers!

The submission of a replication attempt is NOT a prerequisite for participation in the workshop!

However, without at least 2-3 replication attempts, the original plan for the workshop will have to be modified.

Deadline: Register for participation by 31 May 2022 by email to the organizers. If you have completed or attempted a replication, please attach the replication report as a single PDF or the link to the repository containing the materials (including a text document following the template’s report structure!).