Longitudinal assessment of research in GIScience domain shows a positive impact of reproducible research practices
https://doi.org/10.31223/X5RJ3W #GeographicInformationSciences #Giscience #ReproducibleResearch #Reproducibility #ComputationalScience #DataScience #OpenScience #Meta-science #Meta-research #SciencePolicy #OpenAccess
What did we do?
A team of assessors analysed the potential reproducibility based on 200+ papers from the AGILE and GIScience conference series based on four criteria with different levels: input data, methods/analysis/processing, computational environment, and results.
Comp. env could be 👍 / 👎 , the others' levels are:
🇺 undocumented
🇩 documented
🇦 d. + available
🇴 d. + a. + open
Interested in the methods and the raw data? Check out our research compendium at https://github.com/nuest/reproducible-research-giscience-longitudinal-study
What did we do?
A team of assessors analysed the potential reproducibility based on 200+ papers from the AGILE and GIScience conference series based on four criteria with different levels: input data, methods/analysis/processing, computational environment, and results.
Comp. env could be 👍 / 👎 , the others' levels are:
🇺 undocumented
🇩 documented
🇦 d. + available
🇴 d. + a. + open
Interested in the methods and the raw data? Check out our research compendium at https://github.com/nuest/reproducible-research-giscience-longitudinal-study
What did we find?
📈 The assessment shows similar patterns to the results both conference before the AGILE conference started reproducibility reviews based on reproducibility guidelines. in other words, the top rows of both these figures are quite similar. Lower rows in both figures present right-skewed distributions so there is overall improvement, but post-intervention AGILE papers show a more pronounced shift towards higher levels across all criteria. We have statistics for that, too!
We conclude 🏁 in our new preprint https://doi.org/10.31223/X5RJ3W
A shift in Open Science practices can be introduced and observed at a small community conference with the support of community leadership [..]. The [..] #CODECHECK [process] can be considered [..] transferable to other communities. We believe our results provide strong motivation for [..] journals or conferences to [..] introducing reproducibility guidelines for authors and reviewers, and implementing code execution checks [..].
@eartharxiv
📢 The next paper exploring the reproducibility of #GIScience publications is out!
I'm extremely grateful for the continued collaboration with @f_ostermann & Carlos Granell & others, and excited about the results. We collected evidence that the combination of reproducibility guidelines and reproducibility reviews can indeed improve the potential reproducibility of data/software-based research publications.
#OpenScience #AGILEGIS