What is Reproducibility

In the era of ‘questioning everything’ with respect to its impact on neuroimaging analysis reproducibility, we start with a set of petites histoires which take a look at the implications of various choices that researchers routinely make, and often take for granted.

First, let’s set the stage; while there are many definitions around the concept of ‘reproducibility’, I’m a bit partial to the one reflected in the following figure: 

Here, we define a number of concepts that we will return to over and over again in the course of our stories:

• Re-executability (publication-level replication): The exact same data, operated on by the exact same analysis should yield the exact same result. Current publications, in order to maintain readability, do not typically provide a complete specification of the exact analysis method or access to the exact data. Many published neuroimaging experiments are therefore not precisely re-executable. This is a problem for reproducibility.
• Generalizability: We can divide generalizability into three variations:
  • Generalization Variation 1: Exact Same Data + Nominally ‘Similar’ Analyses should yield a ‘Similar’ Result (i.e. FreeSurfer subcortical volumes compared to FSL FIRST)
  • Generalization Variation 2: Nominally ‘Similar’ Data + Exact Same Analysis should yield a ‘Similar’ Result (i.e. the cohort of kids with autism I am using compared to the cohort you are using)
  • Generalized Reproducibility: Nominally ‘Similar’ Data + Nominally ‘Similar’ Analyses should yield a ‘Similar’ Result

We contend that ‘true findings’ in the neuroimaging literature should be able to achieve this ‘Generalized Reproducibility’ status in order to be valid claims.  As generalized reproducibility takes numerous claims and multiple publications in order to be established, most publications, themselves, are reporting what I would call ‘proto-claims’. These proto-claims may, or may not, end up being generalized. Since in our publications we do not really characterize data, analysis, and results very exhaustively, this lack of provenance permits the concept of ‘similar’ to have lots of wiggle room for interpretation (either to enhance similarity or to highlight differences, as desired by the interests of the author). In addition, we, as a community, tend to treat these individual reports, or proto-claims, as if they are established scientific claims (generalizably reproducible), since we do not really have any proper ‘system’ (apart from our own reading of the literature) to track the evolution of a claim.

While much of the work of ReproNim is to help establish easy-to-use end-user tools to exhaustively characterize data, analysis, and results (in order to enhance the community’s ability to explore the ‘reproducibility landscape’ of any given publication and its claims), it is equally important to work on the claims identification and tracking problem so that we can detect when our more ‘reproducible and re-executable’ procedures have established the ‘generalized reproducibility’ of a specific finding.  Our next petites histoires will delve more deeply into the details of neuroimaging analysis ecosystem.