28 Reproducibility Check

When authors share their data and analysis code, the idea is that anyone should be able to run that code on that data and get the results reported in the paper. But sharing data and code in a way that results are reproducible is difficult, and researchers have often not received training in how to share data and code. As a consequence, there are often basic mistakes in public repositories that stand in the way of computationally reproducible results. Automated checks can help researchers identify and fix these mistakes, such as not sharing all files that are loaded in code, using absolute instead of relative filepaths, using values that are not computed in the code that is shared, changing file names so they no longer match the code, or not loading all libraries.

Checking whether data and code is reproducible is tedious. It also often requires repeated back and forths between the person performing the checks, and the researchers sharing data and code. Ideally, checks do not just focus on whether code runs, but also whether all results in a manuscript are reproduced by the code, but this takes a lot of time.

Metacheck’s reproducibility_check module attempts to perform all these checks automatically. When it succeeds, researchers and/or reviewers, data managers, or editors know that data and code is shared in a reproducible manner, and they do not need to perform manual checks. This saves everyone time, and leads to data and code repositories that are more useful for science. Our reproducibility check module is completely code-based. It uses no AI: when it runs the modules it builds on, such as data_check, it turns the use of a large language model off for them, also when you have set llm_use(TRUE). It can run locally on your computer without sending information to an external server.

This chapter starts with the module at work on two versions of the same small study: one where every reported result can be reproduced, and one where several cannot. The sections after the example describe each step of the check in detail.

28.1 Computationally reproducing a simple Stroop experiment

Both papers analyse the same classroom data from Daniël Lakens’s introduction to psychology and technology course. Students named the ink color of words that matched the color (the word “red” printed in red) and of words that did not (the word “red” printed in green), and reported the time they needed in seconds. Both papers report four results: the mean and standard deviation of the reaction times in each condition, the correlation between the reaction times in the two conditions, a paired t-test for the Stroop effect, and a comparison of the Stroop effect between the students tested in 2013 and in 2014.

The good paper, https://github.com/Lakens/stroop_effect_paper, computes every reported number in its analysis code. The manuscript is a Quarto document, so the text prints these numbers directly from the analysis.
The bad paper, https://github.com/Lakens/stroop_effect_paper_bad, has several common mistakes deliberately built in. The results were typed into the text by hand, the shared data contain a value the authors removed before their own analysis without saying so, some reported results are not computed by the code at all, and the code stops with an error before it reaches the last analysis.

28.2 Running the code locally, or using our server on SURF

The reproducibility check runs code that other people wrote. That is not without risk. A script can delete files, read files that have nothing to do with the project, or send information over the internet, by accident or on purpose. By default, execute = TRUE runs each script in a separate R process on your own computer. This prevents a script that crashes from taking your own R session down with it, but the script can still reach every file on your computer, and the internet. That is fine for the two repositories in this chapter, which we made ourselves, but not for code from a repository you have not created, as a bad actor might insert problematic code.

For such code, add sandbox = "docker". Docker is a program that runs software in a container: a closed environment on your computer, separated from the rest of the system. Metacheck runs every script in its own container, and has set up these containers so that the code can do as little harm as possible:

The container has no internet connection while the code runs, unless you allow it with internet = TRUE (see Scripts that download their data). Only the step before it, which installs the packages the code needs, is connected to the internet, and in that step none of the paper’s own code runs.
The code runs on a copy of the repository’s files, in a temporary folder. That folder, and the folder of packages installed for this paper, which is removed after the check, are the only places it can write to, and the files on the rest of your computer are not visible inside the container.
The code runs as an ordinary user without administrator rights, cannot obtain extra rights, and can start only a limited number of processes.
Each script is stopped after a time limit, 10 minutes by default.

No sandbox makes running unknown code completely safe: a flaw in Docker itself could in principle still be exploited. But these settings close off the usual ways in which a script can do damage. Two ways to run the code below lists the exact settings, and compares them with running code without Docker.

Running the check in Docker takes some preparation, and disk space. You need Docker Desktop (on Windows and macOS) or Docker Engine (on Linux), and the space Docker Desktop itself needs. The first time you run the check, Docker downloads Metacheck’s image for R code: R, Quarto, and the roughly 750 R packages that occurred in a scan of thousands of R files shared with psychology papers, together about 4.9 GB. Python code runs in a second image of about 2.7 GB. Because most packages a paper needs are already in the image, they do not have to be installed again for every paper, which makes the check much faster (but at the expense of a large docker image that you install once). The pre-built Docker image below explains how it was built.

If you do not use R, the Docker installation runs Metacheck’s website on your own computer, with the same checks. It needs about 1.7 GB of disk space and a computer with 8 GB of memory, plus Docker. There, the code of a repository runs inside Metacheck’s own container: it cannot reach the files on your computer, but unlike in the sandbox described above, it can use the internet. The packages the code needs are downloaded while the check runs. Running the check on your own computer also means that an unpublished manuscript never leaves it, which matters for editors and reviewers who may not upload a submission to an external server.

The easiest way to try the check is our server on SURF Research Cloud, which runs on the servers of SURF, the cooperative of the Dutch universities, in the Netherlands. You need nothing but a web browser. On Metacheck’s website, upload a PDF under Run Metacheck on a Paper and tick Also run the code in the paper’s repository, or, under Run Metacheck on a Repository, choose a folder or a .zip file, or paste a link to an online repository such as GitHub or OSF; there, running the code is ticked by default. The ResearchBox Check and the TU/e Data Manager Check run the check too. The server already has the most common R and Python packages. It first installs any other packages the code needs from CRAN or PyPI, but not packages from GitHub or other links, and then runs each script for at most 5 minutes, in a closed sandbox without internet access. Uploaded files are deleted as soon as they have been read, and the report is kept on the server for one hour, so save it if you want to keep it.

The server’s capacity is shared by everyone who uses it, so it is meant for checking a paper or a repository now and then. If you want to check many papers, run Metacheck on your own computer, or build your own server on SURF Research Cloud, which researchers at Dutch institutions can do through their institution or with a free NWO grant. The Setting up Metacheck chapter describes all these options.

28.3 Running the check in R

We illustrate the module in R. Metacheck reads a paper from its PDF. We download both PDFs from GitHub, and convert() turns each into a structured file that read() loads into R. If you want to follow along in the Metacheck website, download the two papers from here and here.

download.file(
  "https://raw.githubusercontent.com/Lakens/stroop_effect_paper/main/output/analysis.pdf",
  destfile = "good_paper.pdf", mode = "wb"
)
download.file(
  "https://raw.githubusercontent.com/Lakens/stroop_effect_paper_bad/main/analysis.pdf",
  destfile = "bad_paper.pdf", mode = "wb"
)
paper <- read(convert(file_path = "good_paper.pdf", save_path = "."))
bad_paper <- read(convert(file_path = "bad_paper.pdf", save_path = "."))

Both papers link to their repository. The repo_check module in Metacheck supports over 270 data repositories, and can automatically download files from these repositories.

Running the module is a single line. By default, reproducibility_check only inspects the code without running it. execute = TRUE is needed to actually run the full computational reproducibility check:

mo_good <- module_run(paper, "reproducibility_check", execute = TRUE)
mo_good$traffic_light

#> [1] "green"

cat(mo_good$summary_text)

#> 
#> -  We assessed AND ran 2 R code files for reproducibility (each in an isolated subprocess).
#> -  2 of 2 scripts ran without error when executed.
#> -  5 installable dependencies detected.
#> - 39 statistics stored from the extracted output; 11 of 11 reported tests matched (100%).

The light is green: every script ran without an error, and every result reported in the manuscript was among the results the code produced when it ran. mo_good$table lists each script and what happened when it was run:

mo_good$table[, c("file_name", "outcome")]

#>                 file_name outcome
#> analysis.qmd analysis.qmd  ran_ok
#> clean_data.R clean_data.R  ran_ok

clean_data.R turns the raw data file into the processed data file, and analysis.qmd is the manuscript, whose code chunks contain the analysis. The repository also contains renv/activate.R, a script the renv package adds to a project to set up its package library. The module does not check the contents of this folder, because it knows it is not part of the analysis.

mo_good$match_table shows, for each result reported in the paper, whether it was found in the output of the code, and in which analysis:

mo_good$match_table[, c("reported", "found", "confidence", "analysis")] |>
  knitr::kable()

reported	found	confidence	analysis
M=15.1	TRUE	full	mean by group
M=23	TRUE	full	mean by group
SD=4.1	TRUE	full	sd by group
SD=4.78	TRUE	full	sd by group
r=0.37 t=4.53 df=129 p=<0.001	TRUE	full	Pearson’s product-moment correlation
t=18.04 df=130 p=<0.001	TRUE	full	Paired t-test
Cohen’s dz=1.58	TRUE	full	d_z
M=7.77	TRUE	full	mean by group
M=8.13	TRUE	full	mean by group
t=-0.42 df=120.37 p=0.673	TRUE	full	Welch Two Sample t-test
Cohen’s d=-0.07	TRUE	full	Cohens_d

A result is not compared as single numbers, but as a whole: the t-value, the degrees of freedom, and the p-value of one test have to appear together in the output of one analysis. Any single number will sooner or later turn up somewhere in a large output by coincidence, but several numbers of one test appearing together in one analysis will not.

Making your own results findable

The module can only find a result if the code produces it while it runs, in a form the module recognises. In the good paper, every reported number is computed in a code chunk, and the text of the manuscript only rounds these values. A number that is computed only inside the text, by an inline R expression in Quarto or R Markdown, is not part of the code that is run, so it cannot be found.

The module recognises the results of test functions such as t.test() and cor.test(), of summaries such as sapply(stroop_data[, c("congruent", "incongruent")], mean), and of effect sizes from the effectsize package, also when they are stored in a variable (ttest_result <- t.test(...)). A number computed by hand, such as a Cohen’s d computed as mean(diff) / sd(diff), is found when the code prints it, but not when it is only stored in a variable.

A green light means that the numbers in the paper follow from the shared code and data. It does not mean that the analysis was the right one to perform.

Now let’s see what the module returns if we run it on the bad paper:

mo_bad <- module_run(bad_paper, "reproducibility_check", execute = TRUE)
mo_bad$traffic_light

#> [1] "red"

cat(mo_bad$summary_text)

#> 
#> -  We assessed AND ran 1 R code file for reproducibility (each in an isolated subprocess).
#> -  0 of 1 script ran without error when executed.
#> -  3 installable dependencies detected.
#> - 10 statistics stored from the extracted output; 2 of 6 reported tests matched (33.3%).

The light is red, because the analysis code stopped with an error:

mo_bad$table[, c("file_name", "outcome")]

#>                 file_name outcome
#> analysis.qmd analysis.qmd errored

cat(mo_bad$run_results$error)

#> ! in callr subprocess.
#> Caused by error in `eval(predvars, data, env)`:
#> ! object 'stroop_effect' not found

The code compares the Stroop effect between the two years using a column stroop_effect, but no line of the code creates that column. The authors presumably created it in their R console while they worked, so the code ran on their own computer, but it cannot run for anyone else. This is one of the most common reasons shared code fails.

Before the code was run, the module also changed one line of it. mo_bad$modifications lists every change:

mods <- mo_bad$modifications
mods[mods$change_type == "setwd_removed", c("change_type", "detail")]

#>     change_type                                detail
#> 2 setwd_removed C:/Users/dlakens/Desktop/stroop_study

The code starts with setwd("C:/Users/dlakens/Desktop/stroop_study"), which sets the folder R works in to a folder that only exists on the author’s computer. The module skips this line, and reports that it did, because it would stop the code on any other computer.

Now the comparison between the paper and the output:

mo_bad$match_table[, c("reported", "found", "confidence", "analysis")] |>
  knitr::kable()

reported	found	confidence	analysis
t=18.04 df=130 p=0.25 Cohen’s dz=0.25	FALSE	none	NA
t=-0.42 df=120.37 p=0.673	FALSE	none	NA
M=15.1	TRUE	full	mean by group
SD=4.1	TRUE	full	sd by group
M=23 SD=4.78	FALSE	none	NA
r=0.37	FALSE	none	NA

Only the mean and standard deviation of the congruent condition were found. Each of the other results is missing for a different reason:

The incongruent mean and standard deviation (M = 23.0, SD = 4.78) are not in the output. The output contains the statistics the code did compute:

mo_bad$stat_output[[1]]$long[, c("analysis", "statistic", "value")]

#>         analysis       statistic                value
#> 1  mean by group            mean     15.1102878787879
#> 2  mean by group            mean      22.080446969697
#> 3    sd by group              sd     4.08488216267072
#> 4    sd by group              sd       11.63934788467
#> 5  Paired t-test               t     6.77356838604279
#> 6  Paired t-test              df                  131
#> 7  Paired t-test               p 3.84360296105763e-10
#> 8  Paired t-test mean difference     6.97015909090909
#> 9  Paired t-test        conf.low     4.93450571703671
#> 10 Paired t-test       conf.high     9.00581246478147

The incongruent mean is 22.08, with a standard deviation of 11.64. The reason is in the data. One participant has a reaction time of -99 in the incongruent condition:

bad_data <- read.csv("https://raw.githubusercontent.com/Lakens/stroop_effect_paper_bad/main/stroop.csv")
summary(bad_data$Incongruent)

#>    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
#>  -99.00   19.27   22.64   22.08   26.10   35.13

-99 is a code for a missing value, not a reaction time. The authors removed this participant before computing the results in their paper, but the code they shared does not, so it analyses -99 as if it were a reaction time.

The paired t-test, t(130) = 18.04, is not found for the same reason: the shared code analyses all 132 participants, including the -99, and gets t(131) = 6.77. The reported p = .25 and Cohen’s d_z = 0.25 would not have matched in any case, because neither is consistent with a t-value of 18.04 (the StatCheck and Effect Sizes modules flag both from the text alone).
The correlation, r = 0.37, is reported, but no line of the code computes it.
The comparison between the years, t(120.37) = -0.42, p = .673, has exactly the numbers the good paper reports, so it is probably correct. But the code stopped with an error before it reached this test, so it cannot be verified from what the authors shared.

The three p-values about accuracy (p = .07, p = .34, and p = .000) are not in the table at all. The module only checks a p-value that is reported together with its test statistic. Had they been checked, they would not have been found either: the repository contains no accuracy data.

28.4 What it checks

Where code_check asks “is this code written well?”, reproducibility_check asks a harder question: could this code actually be run on this data — and do the numbers it produces match the ones the paper reports? It is the module that stands closest to the goal of computational reproducibility.

The module works in two phases, and you choose how far to go:

A static phase (always run) that works out everything a reproduction attempt would involve — the dependencies, the file-path rewrites, the run order, the missing inputs — and reports where that attempt is likely to break, without running any code.
An execution phase (opt-in, execute = TRUE) that actually runs the code, either in a subprocess on your own machine or inside a locked-down Docker container, against a throwaway copy of the data, captures the statistical output it produces, and — together with any self-contained JASP/jamovi files in the deposit — lets you check whether the reported results are reproduced.

Both phases cover R code and Python code. Most of this chapter describes R; Python code below describes where Python is handled differently.

Note

This module builds on repo_check, data_check, and code_check, and inherits their live network calls to retrieve the repository files. When you have not run data_check and code_check on the paper first, the module runs them itself. You need an internet connection to run the code below.

28.5 The static phase, without running code

The default is the static phase — it does not run anything. This example reads every repository demopaper() links to, so (like the rest of this chapter) it needs those repositories to be reachable; the output below is shown rather than run live, so the chapter does not depend on every linked repository’s uptime at render time:

paper <- demopaper()
mo <- module_run(paper, "reproducibility_check")
mo$traffic_light
#> [1] "yellow"
cat(mo$summary_text)
#>
#> -  We assessed 6 R code files for reproducibility (static analysis; no code was run).
#> -  3 files appear runnable so far (parses, inputs resolve, placeable in the run order).
#> -  26 files skipped as byte-identical duplicates of another file already in the run (the paper links more than one repository/component).
#> -  2 referenced inputs unavailable (0 withheld due to size).
#> -  6 installable dependencies detected.

The table has one row per R or Python code file, carrying the static assessments as columns (there is no language column; the file extension shows the language):

mo$table[, c("file_name", "parses", "run_order", "order_basis",
             "paths_rewritten", "runnable")] |>
  knitr::kable()

	file_name	parses	run_order	order_basis	paths_rewritten	runnable
test_script.R	test_script.R	TRUE	5	none	0	FALSE
bad.R	bad.R	TRUE	2	none	0	FALSE
bad.Rmd	bad.Rmd	TRUE	3	none	1	FALSE
01.R	01.R	TRUE	1	numeric	0	TRUE
good-example.R	good-example.R	TRUE	4	none	0	TRUE
utils.R	utils.R	TRUE	6	none	0	TRUE

A file is marked runnable only when it parses, all of its referenced inputs resolve (they are present, or produced by an earlier script), and it can be placed in the run order. That is a deliberately strict bar for a static check: runnable = FALSE does not mean the code is broken, it means the module could not confirm, without running it, that everything the script needs is in place. When it is FALSE, the column not_runnable_reason names exactly why — one of parse_error, unplaceable, or missing_input, checked in that priority order — and unresolved_inputs names the specific file(s) that could not be resolved. Two further reasons apply only to an R file that does not parse as R: not_r_content when its content is not R at all (for example a JSON or HTML file saved with a .R extension), and jags_model when it is a JAGS/BUGS model definition. Neither counts as a parse error, because the file is not broken R code, and neither is run. Three more reasons name files that are not run as scripts of their own: defines_functions (a file that only defines functions, apart from library() calls and constants; used_by names the scripts that use it), not_analysis (a file of software rather than of the paper’s analysis, such as a package’s own code or tests, a copy of a tool, a web application, or a script a program wrote; not_analysis_why says which), and needs_a_person (an R file that starts an app, such as a Shiny application). For Python, python2 marks code written for Python 2.

Two further list-columns, reads and writes, carry the basenames each script reads and writes (as repro_file_io() extracted them) directly on the table — useful if you want the code↔︎data provenance for a file without recomputing it yourself:

mo$table$reads
#> [[1]]
#> [1] "file.csv"
#>
#> [[2]]
#> [1] "example.csv"
#>
#> [[3]]
#> [1] "example.csv"  "example2.csv"
#>
#> [[4]]
#> character(0)
#>
#> [[5]]
#> [1] "file.txt"
#>
#> [[6]]
#> character(0)
mo$table$writes
#> [[1]]
#> character(0)
#>
#> [[2]]
#> character(0)
#>
#> [[3]]
#> character(0)
#>
#> [[4]]
#> character(0)
#>
#> [[5]]
#> character(0)
#>
#> [[6]]
#> character(0)

28.6 Worked example: building a small reproducible fixture

To see every static assessment concretely, it helps to build a tiny, self-contained example rather than rely on whatever a real deposit happens to contain. The example below creates two scripts — one that writes an intermediate file, one that reads it back — and a data file.

tmp <- withr::local_tempdir()

writeLines(c(
  'dat <- read.csv("data.csv")',
  'summ <- data.frame(mean_x = mean(dat$x))',
  'write.csv(summ, "intermediate.csv", row.names = FALSE)'
), file.path(tmp, "0_prep.R"))

writeLines(c(
  'summ <- read.csv("intermediate.csv")',
  'print(summ$mean_x)'
), file.path(tmp, "1_analysis.R"))

writeLines("x,g\n1,a\n2,a\n3,b\n4,b", file.path(tmp, "data.csv"))

The rest of the chapter uses these two files, 0_prep.R and 1_analysis.R, to demonstrate each static assessment on its own terms.

28.7 The static assessments, function by function

Each static assessment is an exported helper, so you can run it on your own code independently of the module.

28.7.1 Dependencies: `repro_dependencies()`

repro_dependencies() walks R code and returns the distinct packages it loads, tagged with where each would be installed from — a distinction code_check’s package list does not make. A library() call resolves to CRAN by default; a remotes::install_github("user/repo") call sets the install source of the package the code loads under that repository’s name (with the pinned @ref kept when present); packages that ship with R are tagged base, because a run never needs to install them; and a bundled list of common Bioconductor package names (edgeR, DESeq2, and similar) is checked so those are tagged bioc instead of cran — confirmed as the single largest identifiable cause of install failures in a real corpus run, since a Bioconductor-only package is never on CRAN and install.packages() simply fails for it unless this distinction is made upfront.

code_text <- c(
  "library(dplyr)",
  "library(ggplot2)",
  "remotes::install_github('tidyverse/ggplot2@v3.4.0')",
  "x <- stats::sd(1:10)"
)
repro_dependencies(code_text)

#>   package source                      ref  base
#> 1   dplyr   cran                     <NA> FALSE
#> 2 ggplot2 github tidyverse/ggplot2@v3.4.0 FALSE
#> 3 remotes   cran                     <NA> FALSE
#> 4   stats   base                     <NA>  TRUE

Note that a namespace-qualified call — stats::sd(...) and remotes::install_github(...) above — is picked up as a dependency even without an explicit library() call; the last row shows stats tagged base. repro_dependencies() also accepts a list of per-file character vectors and pools them into one result: when the same package is named both bare (library(pkg)) and via an explicit GitHub source somewhere else in the pooled files, the GitHub row wins, since it carries the real install source.

For R code, the dependency list is deliberately name-only, not version-pinned: static analysis cannot recover which version of a package was used. (Python code is the exception: see Python code for how versions are read from a requirements.txt or pyproject.toml.) This is not a limitation to apologise for — it is the point. A reproduction runs against the current versions of these packages, and if it breaks under current versions, that break is itself a finding: it argues that the authors should have pinned their versions.

That said, reproducibility_check does not have to guess blindly at what “current” means. As of the version documented in this chapter, code_check itself checks whether a deposit declared its environment at all — see Declared R versions below, which explains how this feeds directly into the Docker execution path.

28.7.2 Paths: `repro_rewrite_paths()`

A script reads and writes data by relative path — read_csv("raw/x.csv"), saveRDS(m, "../out/model.rds"). The module runs the code in a copy of the repository as the authors organised it (see Where the code runs), so most such paths work as written. Some do not: an absolute path on the authors’ computer, a path from another folder than the one the script runs in, or the name of a file that is elsewhere in the repository. repro_rewrite_paths() maps each referenced path to a file in a plan, a table with one row per file and the path it has in the copy. Inside the module, that plan is the repository itself, and only a path that does not work is rewritten. The function also accepts another plan, such as the one psychds_check makes for a Psych-DS release, as in the example below.

Matching is by basename: a script’s ../data/x.csv and the repository’s raw/x.csv are the same file seen from different working directories, so the prefix is ignored. A name that has no exact match also matches the one file with the same extension and the same name apart from case, spaces and punctuation, so code that reads relative cover.csv reads relative_cover.csv (repositories such as Dryad change names in this way on deposit); table$names_corrected and $modifications (name_corrected) record such a change. When several files share a basename — a demographics.csv in more than one study — the ambiguity is resolved by study group: the candidate whose target path is in the same study as the script wins. Only when that still leaves more than one candidate is the reference left unrewritten and flagged as ambiguous, rather than guessed. A path is only changed where it stands in quotes in the code.

plan <- data.frame(
  file_name = c("demographics.csv", "scores.csv"),
  target_path = c("study-ex1/data/source-demographics_data.csv",
                  "study-ex1/data/source-scores_data.csv"),
  current_path = c("ex1/demographics.csv", "ex1/scores.csv")
)
repro_rewrite_paths('d <- read.csv("data/demographics.csv")',
                    "ex1/analysis.R", plan)

#>                     ref         basename matched
#> 1 data/demographics.csv demographics.csv    TRUE
#>                                        target ambiguous n_candidates is_call
#> 1 study-ex1/data/source-demographics_data.csv     FALSE            1   FALSE
#>   name_in_code name_in_deposit in_join in_here
#> 1         <NA>            <NA>   FALSE   FALSE

Two refinements worth knowing about if you inspect this output closely:

Genuinely identical plan rows are not treated as ambiguous. If the same physical file is listed more than once in the plan (which happens when a paper links several mirrored repositories), and every candidate row resolves to the same target path, that is not ambiguity — there was only ever one possible answer, and repro_rewrite_paths() recognises this rather than flagging it.
Mirror duplicates are collapsed by content, not by name, when you pass structure_df (the data_check structure table). A paper linking an OSF component and its own “Archive of OSF Storage” snapshot can have the same physical file grouped into two different (and mostly wrong) study labels by upstream heuristics; repro_rewrite_paths() hashes each candidate’s actual bytes first, so byte-identical mirrors count as one candidate rather than several conflicting ones.

28.7.3 Run order: `repro_file_io()` and `repro_run_order()`

Scripts in a repository usually depend on each other, and running them in the wrong order fails for reasons that have nothing to do with reproducibility. repro_file_io() first extracts, for every file, the data it reads, the data it writes, and the scripts it sources:

code_text_list <- list(
  "0_prep.R"    = readLines(file.path(tmp, "0_prep.R")),
  "1_analysis.R" = readLines(file.path(tmp, "1_analysis.R"))
)
io <- repro_file_io(code_text_list)
io

#>      file_name            reads           writes sources source_refs
#> 1     0_prep.R         data.csv intermediate.csv                    
#> 2 1_analysis.R intermediate.csv

repro_run_order() then derives an order from three signals, in decreasing strength:

read-after-write — if one script writes intermediate.csv and another reads it, the writer must run first (a hard data dependency);
source() edges — if A source()s B, B must run first;
numeric filename prefixes — 0_prep.R before 1_analysis.R, and also embedded numbering such as Exp1_02_... before Exp2_1_... (a weak tie-breaker, not a dependency). The numeric key is the vector of every digit run anywhere in the filename, not just a leading prefix, so Exp1_02_preprocessing.R correctly sorts before Exp2_1_import.R.

Python has no equivalent of source() that the module follows, so Python scripts are ordered by the first and third signals only. For R, repro_run_order() also returns every source() link it found, as a data frame in its "source_edges" attribute (from is the script that is sourced, to the script that sources it); when the code is run, the module uses it to run a sourced script only inside the script that sources it (see the outcome run_by_source below).

The scripts are then topologically sorted. A dependency cycle (A reads a file B writes, and vice versa) cannot be ordered and is reported; so is an ambiguous ordering, where nothing — no dependency and no numbering — distinguishes the order of two or more files.

repro_run_order(io)

#>      file_name order depends_on order_basis
#> 1     0_prep.R     1                numeric
#> 2 1_analysis.R     2   0_prep.R  dependency

0_prep.R runs first even though its own name sorts it that way anyway — but critically, the order_basis column reads "dependency", not "numeric", because the real reason is the data dependency: 1_analysis.R reads intermediate.csv, which 0_prep.R writes. order_basis is recorded on the file that has the incoming edge (the dependent file), not the file it depends on — so it is 1_analysis.R’s row that carries "dependency" and names 0_prep.R in depends_on.

28.7.4 Missing inputs: `repro_missing_inputs()`

When a script reads a file that is not present, the reason matters. repro_missing_inputs() cross-references each unresolved read against the file plan and the download records, and classifies it as one of:

withheld_size — the file is in the repository, but was not downloaded because it exceeded a size cap. This is a size-cap issue, not a reproducibility failure: raising max_file_size / max_download_size (in data_check) would fetch it.
in_repo_not_downloaded — listed in the repository but not present locally for another reason.
in_archive — the file is in the repository only inside a .zip or .tar (for example data.zip/data/x.csv), and the code reads it at the path unpacking would give (data/x.csv). The reason reads “only inside data.zip; unpack it so the code finds it”.
absent — not in the repository at all. The script reads a file the authors did not share (or that an earlier, un-captured step was meant to produce). For such a file, similar_to names a file in the repository with a similar name, if there is one, such as the same name with a piece added at its start or end; the report names it, but the module does not use it.

A file that a script writes itself, and then reads back, is not a missing input. Neither is a name the module cannot check before the code runs: a paste() or sprintf() call that holds a variable, a name with a placeholder such as {name}.csv, or a piece of a name such as _field.csv.

These are predictions. With execute = TRUE, a script with a predicted missing input is still run, because the prediction is often wrong (the file may be read only inside if (file.exists(...)), or its name may be built while the script runs), and the run gives the script’s real outcome and error. After a run, a predicted missing input only counts for a script that did not run to the end.

This distinction is the reason the static phase is worth running before a manual reproduction attempt: a run that fails because a 3 GB data file was skipped by the size gate is a completely different situation from a run that fails because the input data were never shared, and only the latter is a genuine openness problem.

28.7.5 Two more static helpers you may not need directly

Two further exported helpers feed the module internally and are rarely called on their own, but are worth knowing about if you are extending the checks:

repro_defined_vars(code_text_list) returns, for each R file, the variable names it assigns at its top level (not inside a function). This is what powers the corrective re-run for R described below: when a script fails with object 'x' not found, the module looks for exactly one other file whose repro_defined_vars() result includes x, and — if there is exactly one — infers that it must run first.
repro_materialize_layout() and repro_write_scripts() build the throwaway copy of the repository the execution phase actually runs against, and write the path-rewritten scripts into it. You will see their effects described below (the copied files, and the output/ and statistical_output/ folders), but you would only call them directly if you were building your own execution pipeline on top of metacheck’s static analysis. repro_write_scripts()’s inject_libs argument adds one or more library() calls to the top of a named R script; the corrective re-run uses it.

28.8 Two ways to run the code

Set execute = TRUE to actually run the paper’s code. A second argument, sandbox, decides where:

# on this machine, in an isolated subprocess (the default)
mo <- module_run(paper, "reproducibility_check",
                 execute = TRUE, sandbox = "process",
                 install_missing = TRUE, timeout = 600)

# inside a locked-down Docker container
mo <- module_run(paper, "reproducibility_check",
                 execute = TRUE, sandbox = "docker",
                 install_missing = TRUE, timeout = 600)

The two options carry genuinely different risk, and the difference is worth understanding precisely rather than taking on faith:

	`sandbox = "process"` (default)	`sandbox = "docker"`
Mechanism	a subprocess on your machine: a `callr` R process for an R script, a `python` process for a Python script	a `docker run` container
Isolates a crash?	Yes	Yes
Isolates the filesystem?	No — the code can read, write, or delete anywhere your R session can	Yes — `--read-only`, with only the throwaway sandbox directory, the paper’s own package library and `/tmp` writable
Isolates the network?	No — the code can reach the internet freely	Yes — `--network none` during the run phase, unless you set `internet = TRUE`
Runs as	your own user account	a non-root user inside the container: on Windows and macOS user 1000; on Linux the user running R (user 1000 when R runs as root)
Requires	the `callr` package (only when the paper has R code); for Python code, the `processx` package and a Python interpreter (`python`, or else `python3`)	Docker installed and running, plus the `processx` package
Speed	fast — no container overhead	pre-built image (see below) makes this close to the process backend for most papers

sandbox = "docker" achieves real containment with these docker run flags, applied to every script individually during the run phase:

--network none --read-only --tmpfs /tmp:exec
--user <uid:gid> --cap-drop ALL
--security-opt no-new-privileges --pids-limit 512 --ulimit core=1
-v <sandbox>:/sandbox -v <library>:/rlib                       (R)
-v <sandbox>:/sandbox -v <library>:/pylib
   -e PYTHONPATH=<the repository's own module folders>:/pylib     (Python)

Network is fully disabled (--network none is left out only when you set internet = TRUE), the filesystem is read-only outside the mounted sandbox (with only /tmp, the sandbox itself, and the library of packages installed for this paper writable), the container runs as a non-root user, all Linux capabilities are dropped, privilege escalation is blocked, the number of processes the container can start is limited, and R cannot write a crash file (on Windows, each R crash in a container wrote a file of about 2.5 GB). Programs in /tmp may run, because code that is compiled while a script runs, such as an rstan model or Rcpp::sourceCpp(), is loaded from there. The library is writable because some packages keep data in their own folder; it belongs to this paper only and is removed after the check. It is kept in a Docker volume rather than in a folder of your computer, because a Windows drive does not tell file names apart by case, and an exFAT drive refuses Linux file permissions; the volume is named after the sandbox and removed with it. The option options(metacheck.docker_lib_volume = FALSE) keeps the library in a folder of your computer instead.

The user the containers run as (<uid:gid> above) depends on the computer. Docker Desktop on Windows and macOS shares folders with a container without asking who owns them, so there the containers run as user 1000. On Linux only a folder’s owner may write in it, so the containers run as the user running R; otherwise, on a computer where R runs as another user (such as a server, or GitHub’s Linux machine), installing packages stopped with “Permission denied”. When R runs as root, the containers still run as user 1000, not as root, and the folders they write in are opened to that user; symbolic links are left alone, so a link in a deposit cannot open a folder outside it. options(metacheck.docker_user = "uid:gid") chooses the user. Before checking any of this, call repro_docker_available() to confirm Docker is installed and its daemon is actually running — reproducibility_check() calls this for you and stops with a clear message if it is not, rather than failing partway through a batch of papers with a cryptic error. When Docker is slow to answer, because other containers are busy, it asks again, waiting up to 30 and then 60 seconds, and when Docker still does not answer, the message says so instead of saying that Docker is not running. A script whose container never started, or that Docker lost while it ran, gets the outcome docker_failed, which says nothing about the script.

repro_docker_available()
#> $ok
#> [1] TRUE
#> $msg
#> [1] ""

Note

The dependency-install phase under sandbox = "docker" runs with the network on (it has to, to reach CRAN or GitHub, or for Python code the package index that pip installs from) but is otherwise hardened the same way — non-root, capabilities dropped, no new privileges. Only the per-script run phase disables the network entirely, since that is the phase running the paper’s own, potentially untrusted, analysis code.

Tip

Working through a whole corpus rather than one paper? Batch mode, described later in this chapter, runs several papers’ sandbox = "docker" checks concurrently instead of one at a time.

28.9 The pre-built Docker image, and why it exists

The first time you think through what sandbox = "docker" needs to do to run a real paper’s code, an obvious problem appears: most papers’ scripts library() several packages that are not part of base R, and a container starts with nothing installed. Naively, every single reproducibility_check() call would need to install a paper’s entire dependency list from scratch, inside a freshly-started container, before it could run a single script — for R packages with compiled C++ code (rstan, lme4, and dozens of others), that can mean minutes of compilation per paper.

To avoid that, Docker containers under sandbox = "docker" default to a pre-built image, ghcr.io/scienceverse/metacheck_r_large:latest, rather than a bare R installation. That image already has R, Quarto, JAGS, Java, the GDAL/GEOS/PROJ libraries for geographic data, and the roughly 750 most common packages a real corpus of psychology-science papers’ code actually uses — found by scanning thousands of real .R/.Rmd/.qmd files for their library()/require() calls and pre-installing everything that showed up often enough to be worth the build cost. When a paper’s dependencies are already in that list — which, for an ordinary tidyverse-and-lme4-shaped analysis, they usually are — the install phase does almost nothing, because requireNamespace() already finds the package. Only genuinely uncommon or paper-specific dependencies still need a real, from-source install. A smaller variant, metacheck_r_small (about 390 packages, each seen in at least three files of that scan, without the geographic libraries), is also published, but reproducibility_check() has no argument to choose it. Python code runs in a separate image, ghcr.io/scienceverse/metacheck_py_small:latest by default — see Python code.

When a package is not in the image, the install looks for it in this order: CRAN, Bioconductor, the latest version in the CRAN Archive (for a package CRAN has removed, such as rgdal, rgeos and maptools, which compile when the image has the GDAL and GEOS header files), and then, in Docker only, a table of known places (such as rethinking on GitHub or INLA’s own repository) and R-universe’s search for that exact name. A package that could not be installed is reported with the reason from the install’s own output. When the code loads a package that is on neither CRAN nor Bioconductor and does not say where it comes from, the report says so. Only the packages of the files that are run are installed, not those of a package’s own code or tests, or of a tool the authors copied into the repository. While it installs, the module shows the package, the count, the time so far and an estimate of the time left.

The container’s home folder cannot be written to, so R’s cache, configuration and data folders are placed in /tmp. This lets a script run renv::restore(), which needs to write its cache. rmarkdown::render() finds the Pandoc program that Quarto bundles in the image, through the RSTUDIO_PANDOC environment variable.

The image, its Dockerfile, the full package list, and detailed notes on how it was built (including some genuinely non-obvious bugs found along the way — a package can report a successful install and still fail to load, because a runtime shared library it needs is missing from a minimal image) are all in a dedicated repository: scienceverse/metacheck_docker_reproducibility. If you want to build your own variant — a smaller image with only the most common packages, or one that adds LaTeX for PDF rendering, or CmdStan for cmdstanr-based Bayesian models — that repository is the starting point, and its README documents exactly which levers actually reduce image size and which ones (surprisingly) do not.

28.10 Declared R versions, and why they matter for Docker

This section applies to R code only; metacheck does not read a declared Python version, and Python code always runs in the pre-built Python image.

The pre-built image is fast, but it was built against one specific R version. A paper that deposited an renv.lock file, or a sessionInfo() text dump, or that pins its environment with groundhog/checkpoint, has told you exactly which R version its authors actually used — and that might not be the version the pre-built image happens to run.

code_check is where this gets detected, as part of its own “Reproducible Environment” check: it looks for an renv.lock file anywhere in the repository (parsing its declared R version and every locked package + version), a sessionInfo()/session_info() dump (matched by filename — sessionInfo.txt, session_info.txt, and similar — or embedded in a README, or pasted into an R script, usually as comments, when it lists packages; the package versions it lists are kept as well as the R version), or an actual groundhog::groundhog.library()/checkpoint::checkpoint() date-pin call in the code, also when the date is held in a variable or given as snapshot_date — a bare library(groundhog) with no pinning call does not count, since loading the package is not evidence anything was actually pinned, and neither does a call in a comment.

cc <- module_run(paper, "code_check")
cc$traffic_light
#> [1] "yellow"

When nothing was declared, code_check’s own report explicitly says so and recommends depositing one — this is itself a piece of reproducibility hygiene worth checking regardless of whether you ever run the code.

reproducibility_check() has a parameter, docker_use_declared_version, that decides what to do with this information:

# default: always use the fast pre-built image, regardless of what the
# paper declared
mo <- module_run(paper, "reproducibility_check",
                 execute = TRUE, sandbox = "docker")

# opt in to matching the paper's own declared R version exactly
mo <- module_run(paper, "reproducibility_check",
                 execute = TRUE, sandbox = "docker",
                 docker_use_declared_version = TRUE)

By default (docker_use_declared_version = FALSE), the module always uses the pre-built metacheck_r_large image, prioritising speed. If the paper did declare a version, you get a warning naming the mismatch and explaining how to opt out of it — this is a deliberate design choice: silently running a paper’s code against a different R version than the one its authors used could, in principle, change results, and that substitution should never happen invisibly.

Warning message:
reproducibility_check(sandbox = "docker"): this paper declared R version
4.3.1, but docker_use_declared_version = FALSE (the default), so the run
uses the pre-built metacheck_r_large image's own R version instead, for speed.
Set docker_use_declared_version = TRUE to match the paper's declared
version exactly -- slower, since every dependency then installs from
scratch rather than using the pre-built image.

Setting docker_use_declared_version = TRUE switches to a bare rocker/r-ver:<declared version> image with nothing pre-installed — genuinely slower, since every dependency now installs from source, but it matches the R version the paper’s authors used. When a paper declared no version at all, this option falls back to bare rocker/r-ver:latest rather than silently reusing the pre-built image either — if you explicitly asked for version-matching, getting the fast pre-built image back through the back door when there was nothing to match against would defeat the point of asking. For an R version before 4.0, whose images name MRAN, a copy of CRAN that closed in 2023, the packages come from the same dated copy of CRAN at Posit.

28.10.1 Recorded package versions

The R version is one thing; the versions of the packages are another. With recorded_versions = TRUE (and execute = TRUE), when the repository records package versions — in an renv.lock file, in a sessionInfo() or session_info() record, or as a date in a groundhog.library() or checkpoint() call — the R packages the scripts load are installed at those versions:

mo <- module_run(paper, "reproducibility_check",
                 execute = TRUE, sandbox = "docker",
                 recorded_versions = TRUE)

This works under both sandbox settings, and also when you did not set install_missing (see Options). Only the packages the scripts load are installed at their recorded version; a loaded package the record does not list gets its current version, and the packages these depend on, and those that come with R, are not at their recorded versions. The R version is not changed, so an old package may not install, or may behave differently; a warning and the report say so. After installing, the module checks in a new R process that each package loads: an old version next to current packages can keep one from loading, as a current ggplot2 next to a recorded scales did. A recorded version that cannot be installed, or that keeps a package from loading, is replaced by the current one, and the report names it. $recorded_versions holds which recorded versions the run used.

recorded_versions = FALSE is the default, because on 16 repositories this took about 2.7 times as long, and no failing script started working. When scripts fail and the repository records versions, the report, the summary and the checklist recommend a new run with recorded_versions = TRUE. Recorded versions are not used when several papers run as one; run the papers one at a time, or in batch mode.

When packages are installed before the run, or the run is in Docker, where the network is off, a script’s calls to groundhog, checkpoint and renv would fail, so they are replaced: groundhog.library() becomes library(), and renv::restore() and the like become a statement that does nothing. $modifications records this as env_call_replaced, and the report says so.

28.11 Checking many papers at once: batch mode

Everything so far runs one paper through reproducibility_check() at a time. If you are working through a whole corpus — a paperlist of dozens or hundreds of papers — calling it once per paper in a loop means only one Docker container is ever running: paper 2 does not start until paper 1’s containers (install, then run) have both finished. For sandbox = "docker", where each paper’s own work is a slow, external docker run rather than in-memory R computation, that is time left on the table on any machine with more than one CPU core to spare.

Passing a paperlist together with workers > 1 (and execute = TRUE, sandbox = "docker") switches to batch mode: up to workers papers run their Docker checks at the same time, and the moment one paper finishes, the next paper in the list starts — never fewer than workers running while papers remain queued, never more.

papers <- papers_load("psychsci", cache = TRUE)[1:20]

mo <- module_run(papers, "reproducibility_check",
                 execute = TRUE, sandbox = "docker",
                 workers = 4, timeout = 600,
                 results_dir = "repro_results")

A few things are worth understanding about how this actually runs, rather than treating it as a black box:

Each paper runs in its own background R process (via the callr package — install it if it is not already present), not in your current R session. This is deliberate, not an implementation detail you can ignore: reproducibility_check() reads and restores a few process-wide settings during a single-paper call (llm_use(), an internal option data_check/code_check share for API-limit handling), and those are not safe for several papers to set and restore concurrently in one shared session. A separate OS process per paper sidesteps the problem entirely — one paper’s settings can never be observed by another’s. Every metacheck.* option you have set in your own session (for example the cache directory) is copied into each background process, so each paper’s run uses the same settings yours would.
Every other argument still means exactly what it means for one paper. timeout, install_missing, docker_use_declared_version, cache, and everything else in Options apply individually to each paper’s own run; workers only controls how many of those individual runs are in flight together.
Container resource limits scale with workers automatically. Each container is capped at 1/workers of the host’s CPU and memory (--cpus, --memory, in addition to the --pids-limit 512 every container already had) — so four concurrent papers cannot collectively claim more than the whole machine, and one runaway script cannot starve the other three. You do not set this yourself; it is computed from workers and the host’s own detected CPU count and RAM.
results_dir writes each paper’s full result to disk as soon as that paper finishes — one <paper_id>.rds file per paper, in the same format capture_module_tables() uses elsewhere in Metacheck for a corpus build. This matters for a batch that might take hours: if it is interrupted partway through (a crash, a machine restart, you stopping it deliberately), every paper that had already finished is safely on disk, not lost along with the ones still running. Read them back with metacheck:::collect_module_tables("repro_results", "reproducibility_check") — both capture_module_tables() and collect_module_tables() are internal helpers, not exported, so they need the metacheck::: prefix rather than a plain call.

mo$summary_table[, c("paper_id", "repro_ran_ok", "repro_deps")] |> head()
mo$summary_text
#> Ran the reproducibility check on 20 papers (up to 4 at a time): 17 ok, 2 red, 1 error.

The returned object has the same shape as a single-paper call, just with one row per paper in $table and $summary_table (tagged by paper_id, exactly like every other module that accepts a paperlist), plus one extra element: $per_paper, a named list holding each paper’s own complete, untruncated module result — the full $run_results, $install_results, $stat_output, and $report you would get from calling reproducibility_check() on that paper alone.

Note

Batch mode is specific to sandbox = "docker". Passing workers > 1 with sandbox = "process", with execute = FALSE, or with a single paper does nothing different — those paths are already fast enough, or already isolate only at the crash level rather than the process level, that running several at once is not worth the added complexity of a background-process pool. workers is simply ignored in those cases rather than raising an error, so a script that loops over papers with mixed settings does not need to branch on it itself.

28.12 Running the code: what happens during execution

Whichever sandbox you choose, the mechanics are the same:

copies the repository’s files into a throwaway temporary directory (see Where the code runs below), so the run never touches your real archive or the download cache. The files the code names are downloaded with the others, also when download would leave them out;
writes the path-rewritten scripts into that copy, so their reads and writes resolve;
optionally installs the declared dependencies into a throwaway library (install_missing = TRUE, or recorded versions, see Recorded package versions), never your main R library — under sandbox = "docker" this is always into a throwaway library for the container, since a container has no access to your host library at all. Python packages are installed with pip into a throwaway folder in the same way (with pip install --target under sandbox = "process", and with pip install --prefix inside the container under sandbox = "docker", so that pip uses the packages the image already has). install_timeout limits the time for installing, R and Python together;
runs each script in run order, each in its own subprocess or container with a per-script timeout, capturing the outcome and the console output. An R script that another script runs with source() is not run on its own, but only as part of the script that sources it, and the same holds for a Python file that other files import. Every script that parses is run, also when an input it seems to need was predicted to be missing (see Missing inputs).

mo <- module_run(paper, "reproducibility_check",
                 execute = TRUE, install_missing = TRUE,
                 timeout = 600, keep_sandbox = TRUE)

28.12.1 Where the code runs

The module always runs the code in a copy of the repository as the authors organised it. Earlier versions could also run it in a reorganised Psych-DS copy, chosen with a layout argument; that argument is gone, because a run in a reorganised copy says little about whether the authors’ own deposit works.

Every file the run has (every file of a local folder; for a repository, every file that was downloaded, see download) is copied to its own path, so a path that works in the repository works in the copy. Each script runs from the folder where its paths work, which is not always the top of the repository, since the folder the authors ran it from is not recorded. Only a path that cannot work as written is rewritten: an absolute path on the author’s computer or a path out of the repository, to the file with that name; a path that counts from the script’s own folder (source("helpers.R"), "../data/x.csv") or from its RStudio project’s (.Rproj) folder, to the file’s path from the top of the repository; a path to a file at the top of the repository, from a script that runs in a folder below; and a bare file name read directly, as in read.csv("x.csv"), to the one file of that name in a folder below. A path that is the one argument of here() counts from the folder here() starts in. A Windows path ("Data\\x.txt") gets / between its folders, since on Linux a backslash is part of a file name. This also applies to files the code makes: a script in Study 2/ writing "Data/processed.csv" writes into Study 2/Data/. Any other relative path is left as written, also when other files share its name, since code commonly joins it with a folder of its own. A script’s writes stay where the code puts them, so a later script reads the file where it expects it (only a write to an absolute path or out of the repository goes to output/), and the copied files are writable, so a script can save over a file the repository holds. The folder of a write is made when it does not exist, also when the folder is held in a variable (dirF <- "../Figures/" with ggsave(paste0(dirF, "plot.png"))).

module_run(NULL, "reproducibility_check", local_path = "path/to/folder",
           execute = TRUE)

28.12.2 Time limits

timeout limits each script (600 seconds by default). total_timeout limits all scripts together, R and Python: by default one hour (3600 seconds), counted from when the first script starts. Use NULL or Inf for no limit, or give a clock time (POSIXct) by which all scripts must be done, such as a server’s job limit minus the time its report needs. Once the time has run out, a running script is stopped and no further script is started, so a package with many slow scripts still gets a report. A script that was not started gets the outcome skipped_time_limit, which the report names, and the light is then at least yellow. A script that a later run (see Running scripts again below) was to run with a change, but which had no time left, is also skipped_time_limit, not the first run’s failure that the change was meant to fix. In the first run, a script stopped by total_timeout is timed_out.

To find the statistical results in what a script printed, the module searches only the first 20,000 lines of each script’s printed output, and the report says when it stopped there. The search reads about 700 lines a second, so a script that prints a growing table in a loop could otherwise take longer to search than to run. Set another number with options(metacheck.repro_output_max_lines = ...).

28.12.3 Scripts that download their data

Some scripts download their data while they run. In Docker the network is off while the code runs, so such a script stops, with the outcome needs_internet. internet = TRUE (or options(metacheck.repro_internet = TRUE)) lets the scripts use the internet, under sandbox = "docker" only. What they download follows the limits for the repository’s own files: a script is stopped, with the outcome download_too_large, when the paper’s scripts together have downloaded more than max_download_size, or when a file larger than max_file_size appears. The report then says that the scripts could use the internet, so what they downloaded may differ from what the authors had. internet = FALSE is the default, because nobody has checked the code before it runs. With sandbox = "process" scripts always have the internet, without these limits.

28.12.4 Outcomes

Each script gets one outcome, in the outcome column of $table and $run_results:

Outcome	Meaning
`ran_ok`	the script ran to completion without error
`errored`	the script raised an error (captured, with the message and output)
`timed_out`	still running at the `timeout` cutoff — not a failure, just slow; raise `timeout` for legitimately long analyses (e.g. Bayesian sampling)
`not_parsed`	not run, because the script does not parse (for Python, checked with Python’s own parser)
`python2`	Python code written for Python 2 (a `print` or `exec` statement, `except E, e`, or tabs mixed with spaces). Under `sandbox = "docker"` it runs with Python 2.7, in the official image `python:2.7-slim` (option `metacheck.docker_image_py2`; `FALSE` turns this off), with, when packages are installed, the last versions of its packages that support Python 2, and then gets the outcome of that run. The code is never converted, since that can change results
`dependency_unavailable`	not run (or failed immediately), because a package the script needs is missing: when packages are installed, it could not be installed from any source the install tries (see The pre-built Docker image); when they are not, it is simply not installed on the computer or in the container that ran the script. Either way this is a limitation of where the code ran, not a defect in the paper’s code. For Python: the script stopped with `No module named ...` for a package `pip` could not install, or that was not installed
`not_downloaded`	the script stopped on a file the repository lists but that was not downloaded, such as one over `max_file_size`. It could not be tried, so it is not counted as a script that did not reproduce
`needs_a_person`	the script stopped because it asks a person to choose a file or answer a question, or because it is a program that needs arguments from the person who runs it (R’s `commandArgs()`, Python’s `sys.argv` or `argparse`), or it starts an app, a window or a server, which would wait for a person until the time limit; such an app is not run
`needs_internet`	the script stopped because the network is off in a Docker run, or, with `internet = TRUE`, could not reach what it asked for
`download_too_large`	the script was stopped for downloading more than the limits allow (see Scripts that download their data)
`out_of_memory`	the script, or a program it started (such as a compiler), ran out of the memory each script may use
`docker_failed`	the script’s container never started, or Docker lost it while it ran; this says nothing about the script
`skipped_time_limit`	not started, or, in a later run, stopped early, because `total_timeout` had run out (see Time limits)
`run_by_source`	an R script that another script runs with `source()`. It is run only inside that script, because on its own it lacks the packages and objects the sourcing script sets up first; its result is part of that script’s result. A sourced script that does not parse keeps its own outcome instead
`run_by_import`	a Python file that other files import, such as `utils/data_handling.py`: a module that defines functions for the scripts that import it, which fails or does nothing on its own. It is not run, and not counted among the scripts that ran. A file with an `if __name__ == "__main__":` block still runs. Only `.py` files count as imported
`defines_functions`	a file that only defines functions (apart from `library()` calls and constants); it is used by the scripts that load it (`used_by` names them), not run on its own
`not_analysis`	a file of software rather than of the paper’s analysis: a package’s own code, tests or documentation, a copy of a tool, a web application, a script a program wrote (`not_analysis_why` says which). It is not run and not counted as a script
`jags_model`, `not_r_content`	a JAGS or BUGS model saved with an R extension, which another script reads, or a file with an R extension that is not R at all; not run, and not failures

dependency_unavailable is deliberately excluded from forcing the traffic light red, unlike errored/timed_out — a package that has genuinely vanished from every source metacheck tries is not evidence the paper’s own analysis is broken. not_downloaded, needs_a_person, needs_internet, download_too_large, out_of_memory, docker_failed and skipped_time_limit do not make the light red either, but keep it from being green.

For each script that did not run to the end, the reason column of $table and $run_results says why, in one word, taken from the error that stopped the script (not from a warning before it): file (a file it could not open), package (a missing package), undefined (an object or function that was never defined), syntax, environment (something the computer that ran it lacks, such as a program the code starts, a screen, or memory; the report names a missing program), internet, download, interactive, time, no_message (the script stopped without a message), or other. $summary_table counts the scripts per reason. The error_message column of $table gives the error itself, in at most 300 characters, for R and Python scripts: the error line of the script’s output, as the report shows it (which names a package that is not there), with the address the script asked for when it needed the internet and the error line names none, or the time limit it reached; when there is no error line, the last line of the output. For example: Error in .Primitive(":")() : 0 arguments passed to ':' which requires 2.

28.12.5 Running scripts again

Two R errors lead to one corrective re-run. An error whose message is object '...' not found usually means the script expects a variable another script defines. When exactly one other script in the repository defines that variable — checked via repro_defined_vars() — the module places that script first (adding a “definer runs first” edge and re-sorting), and, because each script has an R session of its own, runs the defining script at the start of the failing script’s session (recorded in $modifications as loaded_first). Authors often run scripts one after another in one session, so that one script’s objects are there for the next. An error whose message is could not find function "...", for a function no script of the paper defines, usually means a package was never loaded. The module then adds library() calls to the top of the failing script: all the packages the project’s other scripts load, when one of them provides the function (such a script was usually written to run after those scripts, in the same R session); otherwise the one package, among the paper’s own dependencies and a short list of very common packages, that exports the function; otherwise, for %>%, magrittr. Each added call is recorded in $modifications, and a script that then runs is reported as reproducible only after adding library() calls it should make itself. This second run runs only what it changes: the scripts that got library() lines or another script to load first, the scripts the new order moves, and the scripts that read what those write. An unchanged script keeps its first result, so a script that ran out of time does not use the paper’s time a second time. These two repairs are decided only once, from the errors of the first run: a second such error after that is accepted as the final result rather than chased indefinitely.

Some scripts then run once more, each for a reason of its own:

A script that stopped on a default of R that has changed since the code was written — before R 4.0, read.csv(), read.table() and data.frame() read text as factors — runs once more with the old default set at its top (old_default), and so do the scripts that stopped on a file it writes.
A file a script could not open, which the repository lists but which was not downloaded, is downloaded then, and the scripts that stopped on it run once more.
A script that stopped on a file another script wrote later in the run runs once more (run_again): a name built while a script runs, such as paste0("input", i, ".txt"), cannot be seen before the run, so the order missed it. A file written in another folder is copied to where the script looked (output_copied).
A script that stopped on a file the repository holds in another folder gets a copy of that file where it looked, and runs once more (file_placed). This covers a path built while the script runs, with file.path(), paste0() or a variable, which cannot be rewritten beforehand.

A script runs once more each time it stops on a file it did not stop on before, up to five times: a figure script that loads the files of four analysis scripts stops on each of them in turn. These later runs apply to R and Python scripts alike, except the one with R’s old defaults. A Python script that fails with a NameError is not run again; see Python code.

Note

Two habits in shared code are handled automatically. A setwd() call in R, or an os.chdir() call in Python — which hardcodes where the code must run and would override the sandbox — is replaced before the script runs by a statement that does nothing (invisible(NULL) in R, pass in Python), with the original call kept in a comment beside it, so a block that held only that call still has a statement. This also applies to base::setwd() and to chdir() under any module name (from os import chdir, or o.chdir() after import os as o). Each change is recorded in $modifications, and the report notes that the script is “reproducible after ignoring a setwd() that should not have been in the code” (or an os.chdir()). An SPSS data file (.sav/.zsav/.por) with no accompanying syntax (.sps) turns the light red, because that analysis cannot be reproduced from the deposit; the same is true for a Stata .dta file with no .do. This holds whatever language the paper’s code is in, also when the paper has only Python code. The report recommends depositing analyses as jamovi or JASP files instead, since those bundle data and analyses together and need no code to be re-run at all.

Note

Byte-identical duplicate files are only run once. A paper that links several repository components mirroring the same materials — a “live” OSF component, an “Archive of OSF Storage” snapshot, a view-only anonymised link — would otherwise have code_check list the same script once per mirror, and the module would run it two, four, or more times for no reason. reproducibility_check hashes each resolved script’s actual content and drops exact duplicates before running, naming which kept file each dropped one mirrors in the report.

28.13 Python code

Python code goes through the same two phases as R code, and the report describes it in its own section, headed “Python code”, after the R sections. A .py file is Python code, and so is a Jupyter notebook (.ipynb) or Quarto document (.qmd) whose code is Python (code_check reads a notebook’s declared kernel, and a Quarto document’s declared engine or else its first code chunk). The code in a notebook or Quarto document is taken out of it first, so that python runs the code and not the notebook’s JSON text or the document’s Markdown.

The static phase differs from R in three ways:

Syntax check. Each file is read by Python’s own parser, which checks the syntax without running the code. This needs a Python interpreter (python, or else python3) on your computer; without one, every file is counted as parsing and the report says that the check could not be done. Code written for Python 2 (for example print "x") gets the outcome python2 instead of counting as a parse error, and is run with Python 2.7 under sandbox = "docker" (see Outcomes).
Dependencies. repro_dependencies_py() reads the packages from the import lines. Modules of Python’s standard library are tagged base, the rest pypi. When the name a package is imported under differs from the name pip installs, the pip name is reported: sklearn becomes scikit-learn, cv2 becomes opencv-python, PIL becomes Pillow, yaml becomes PyYAML, bs4 becomes beautifulsoup4, and so on for about twenty such names. Unlike for R, versions can be known: when the repository has a requirements.txt (or, if not, a pyproject.toml), the requirement it states for a package is kept in ref and used when installing it.
Run order. Python scripts are ordered by read-after-write and by the numbers in their filenames only.

code_text <- c(
  "import numpy as np",
  "from sklearn.linear_model import LinearRegression",
  "import os"
)
repro_dependencies_py(code_text, manifest_text = "scikit-learn==1.3.0")

#>        package source                 ref  base
#> 1        numpy   pypi                <NA> FALSE
#> 2 scikit-learn   pypi scikit-learn==1.3.0 FALSE
#> 3           os   base                <NA>  TRUE

With execute = TRUE:

Each Python script runs whole, as its own python process. With sandbox = "process" this needs the processx package and a Python interpreter on your computer; if no interpreter is found, or installing or running stops with an error, the Python files are reported as not run, with the reason, and the R results are still returned. With sandbox = "docker", each script runs in its own container from the image ghcr.io/scienceverse/metacheck_py_small:latest, which has about 45 common packages (numpy, pandas, scipy, matplotlib, seaborn, scikit-learn, statsmodels, and others) already installed. reproducibility_check() has no argument to choose another Python image, but the option metacheck.docker_image_py names one, such as "ghcr.io/scienceverse/metacheck_py_large:latest", which adds about 70 packages and torch and tensorflow for ordinary processors; Metacheck’s server uses that image. repro_install_deps_py_docker() and repro_run_scripts_py_docker(), called directly, also take an image argument.
With install_missing = TRUE, packages are installed with pip into a throwaway folder, never into your own Python installation: with pip install --target under sandbox = "process", and with pip install --prefix inside the container under sandbox = "docker", where pip also looks in PyTorch’s own collection of versions for ordinary processors.
A script finds the repository’s own modules: the folder that holds each module a script imports, also the modules the files it imports need, is added to the places Python looks, so from maad import sound works when the repository has a maad/ folder.
Because each script runs in its own process, a name (a variable, function, or class) that one script defines is never available to another, whatever the order. A script that fails with a NameError is therefore not re-run; the report names the script or scripts that define the missing name instead. The later runs for files (see Running scripts again) do apply to Python scripts.
What a script prints is read with the same reader used for notebook outputs. A printed scipy result such as TtestResult(statistic=4.89, pvalue=6.75e-05, df=22) becomes a t-statistic, a p-value, and degrees of freedom, which are compared with the paper together with the R output (see Matching the reported results against the output). Only the printed text is read: unlike for R, the result objects themselves are not recorded, so a value a script computes but never prints is not seen.

28.14 What if there’s no R or Python code at all?

Not every deposit has R or Python code to run, and the module handles several of those cases explicitly rather than falling through to a generic “nothing to check” result:

A JASP (.jasp) or jamovi (.omv) file, with no R or Python code anywhere — these formats bundle the data and the analyses and the rendered results together, so they are reproducible from the file itself. The module still extracts and matches their statistics (see below); the traffic light is yellow, not na, because there is a real, substantive finding to report.
Stata .do files with a .smcl output log, but no separate .dta — the .smcl log both recovers the executed syntax (as a synthetic .do file code_check can check) and carries the results it printed, extracted the same way as JASP/jamovi output.
Stata syntax with no data and no output at all — named explicitly in the report as "Stata code without data or output", with the light set to info, because this is real, actionable guidance (deposit a .smcl log) rather than an empty check.
The same for SPSS syntax with no data or output, SAS code, MATLAB code, or any other language code_check recognises but this module does not execute: each gets its own named block explaining specifically what is missing, rather than one generic message.
Genuinely nothing at all — no R or Python code, no self-contained output, no other-language code either — is the only case that returns na.

When a paper does have R or Python code, the same findings about SPSS/Stata files, self-reproducible output, and code in other languages are added to the report after the code sections (under the heading “All code and output” when there is Python code).

# a paper whose only "code" is a jamovi file
mo <- module_run(jamovi_only_paper, "reproducibility_check")
mo$traffic_light
#> [1] "yellow"
mo$summary_text
#> - We assessed 1 self-reproducible output file (JASP/jamovi/SPSS-Viewer/Mplus) instead of R code.
#> - 47 statistics stored from the extracted output.

28.15 Extracting the statistical output

Running the code is only half of reproducibility — the other half is the numbers it produces. Several kinds of deposit already contain, or can be made to produce, a paper’s statistical results, and the module extracts all of them into one common, structured form via read_stat_tables():

JASP (.jasp) and jamovi (.omv) files are self-contained: they bundle the data and the analyses and the rendered results. These are reproducible from the file itself — no code needs to run.
SPSS Viewer files (.spv) likewise carry their own results structure, and — since the .spv’s own structure XML records the exact SPSS syntax that produced each table — the module also recovers that syntax as a synthetic .sps file, checked by code_check like any other syntax file.
Stata output logs (.smcl) are Stata’s own rendered output, echoing every command that was run; the module both extracts the printed results and recovers the commands as a synthetic .do file.
Mplus output (.out) always self-documents: its own “INPUT INSTRUCTIONS” section holds the exact syntax that produced it, so there is no “data without syntax” failure mode for Mplus the way there is for SPSS and Stata.
Jupyter notebooks (.ipynb) save each code cell’s output into the file, so a Python analysis’s printed statistics are recoverable with no Python installed and nothing re-run. (With execute = TRUE, the notebook’s code is also run, as described in Python code.)
Python scripts, when executed, have their printed output read the same way as a notebook’s saved output.
R scripts, when executed, print their results to the console (summary(lm(...)), t.test(...), an anova table). The module parses that captured output into the same structured form, and — where possible — also captures the actual R objects a script’s statements return (an htest, a summary.lm), not just their printed text, so a statistic’s exact value and identity survive even when the console text alone would be ambiguous (the same "W" is Shapiro–Wilk’s after shapiro.test() but the rank sum after wilcox.test()).

Every extracted statistic is typed with the STATO ontology (the Statistical Methods Ontology), so a t-value carries the accession for “t-statistic”, a p-value the accession for “p-value”, and so on; a statistic with no STATO class keeps its own column header as a plain label, so nothing is dropped.

# read the result tables out of a jamovi or JASP file directly
tables <- read_stat_tables("analysis.omv")
long   <- stat_results_long(tables, paper_id = "example",
                            source_file = "analysis.omv")
head(long[, c("source_file", "analysis", "statistic", "stato_label", "value")])

stat_results_long()’s full output has these columns: paper_id, source_file, test_id, result_id, analysis, table_title, row_label, statistic, stato_label, stato_iri, value, and model_ref (the last linking a captured statistic back to the fitted-model object it came from, when one exists — the same mechanism that makes anova(m) and summary(m)$coefficients findable as describing the same underlying model even though neither call names the other).

The full set of results for a paper is also written to a structured JSON document via stat_output_json() — one per source file, with a schema, schema_version, paper_id, source_file, source_format, and an analyses array of typed results. This is worth being precise about: it is a metacheck-native schema, not ISA-JSON. An earlier version of the module modelled this document on the ISA (Investigation/Study/Assay) vocabulary from the life sciences, but that forced statistical results into containers (“a t-test result as a Material”) the vocabulary was never designed to hold — a repurposing metacheck itself invented, not an established convention. The current schema drops that borrowed vocabulary entirely in favour of a flat, self-describing document with a small set of named sections, in the spirit of formats like Psych-DS rather than conforming to an external schema that never quite fit.

28.16 Matching the reported results against the output

The final question — the one reproducibility is really about — is whether the results the paper reports appear in the results the analysis produces. match_reported_output() answers it.

A reported statistical test is not a single number; it is a multi-component statement, for example “M = 1.93, SD = 0.76, W = 183.5, p = .791, rb = -0.16”. Metacheck’s text extractor records each of those numbers in the paper’s $eq table, grouped so that the numbers from one reported test stay together. match_reported_output() recomposes each reported test from its components, then checks whether those component values co-occur in a single analysis of the extracted output.

As the example at the start of this chapter showed, matching whole tests, rather than lone numbers, prevents a value from being found by coincidence.

Inside the module, this comparison is made once, after both the R and the Python code have run, so it uses the output of both languages together with any self-reproducible output. A printed scipy result names its values statistic and pvalue, which say nothing about the kind of test, so the kind is taken from the result’s class name: TtestResult gives a t-statistic, F_onewayResult an F-statistic, PearsonRResult Pearson’s r, and Chi2ContingencyResult a chi-squared statistic with its degrees of freedom (dof). spearmanr() and kendalltau() return the same class, so their coefficient is left without a type.

m <- match_reported_output(paper, mo$stat_output)
attr(m, "summary")
#> $n_tests   19
#> $n_found   14
#> $n_full    11   # every component of the test co-occurs in one analysis
#> $n_partial  3   # most components matched
#> $n_missing  5

Each row of the result is one recomposed reported test, with where it was found. The full column set is text_id, grp_id, reported, n_components, n_matched, found (logical), match_values, not_matched, source_file, analysis, confidence ("full", "partial", or "none"), and plausible_split:

m[m$found, c("reported", "n_matched", "confidence", "source_file", "analysis")]
#>                      reported n_matched confidence          source_file                 analysis
#>  M=1.93 SD=0.76 W=183.5 ...           5       full  experiment_1.omv   One Sample T-Test
#>  p=0.001 d=0.46                       2       full  across_all.omv     Paired Samples ...

So a reported test is not merely flagged as “found” — the module tells you which deposited file and which analysis produced the matching numbers, which is exactly the provenance a reader needs to trust that the paper’s results are reproducible.

By default, min_components = 1, so even a lone reported statistic — a bare correlation coefficient with no accompanying test statistic — is checked. This is a change from an earlier version of the module, which defaulted to requiring at least two components before treating something as a matchable “test” at all; if you want that stricter behaviour, set min_components = 2 explicitly.

plausible_split deserves a word of explanation, since it is easy to misread. Some reported values only match after being separated from the other values they were originally reported alongside — a mean split from its own confidence interval, say. plausible_split marks whether that separation is well-supported: TRUE when the split pieces still trace back to the same underlying variable via the output’s own row labels, FALSE when no such link was found. A FALSE here is not necessarily wrong, but it is worth checking manually against the source_file/analysis columns before treating it as a confirmed match; NA means the value was never split in the first place, so the question does not apply.

If your extracted paper has any results tables (not just narrative text), pass include_tables = TRUE to also check statistics reported only in a table’s cells against the extracted output.

28.17 The traffic light and what you get back

Light	Meaning
green	every script parses, its inputs resolve, the run order is unambiguous, no path rewrite is ambiguous, and (when executed) nothing errored
yellow	some obstacle short of a full reproduction failure — a parse error, an unresolved input, an ambiguous path or order, a self-reproducible output file present with no R or Python code to run, or (when executed) a script that could not be tried or completed for a reason that says nothing about the code, such as `not_downloaded`, `needs_a_person`, `needs_internet` or `skipped_time_limit`
red	a reproduction failure: code that errored or timed out when run, missing inputs that are entirely explained by size withholding, or SPSS/Stata data deposited without its syntax
info	no R or Python code and no self-contained output, but the module found something specific and actionable to say (e.g. Stata syntax with neither data nor an output log)
na	genuinely nothing to check — no R or Python code, no self-contained output, no other-language code either

The static-only red condition is worth stating precisely, since it is easy to over-read: it fires when there are missing inputs and every single one of them is classified withheld_size — not merely when some inputs are size-withheld. Once code is actually executed (execute = TRUE), an execution error or timeout overrides the static signal and forces red regardless — a script that crashes on its own analysis is a genuine reproduction failure, whatever the static assessment said.

R code and Python code each get their own light by these rules, and the paper gets the more serious of the two (red before yellow before green). SPSS or Stata data without syntax then turns the paper’s light red, whatever language its code is in.

Element	What it contains
`$table`	one row per R or Python file: `file_path`, `parses`, `run_order`, `order_basis`, `depends_on`, `paths_rewritten`, `paths_ambiguous`, `setwd_removed` (for Python, the number of `os.chdir()` calls removed), `runnable`, `not_runnable_reason`, `unresolved_inputs`, `reads`, `writes`, `names_corrected`, `used_by`, `not_analysis_why`, `paper_link`, `repo_name` (the repository the file comes from, when a paper links several), and (when executed) `outcome`, `error_type`, `undefined_var`, `reason`, and `error_message` (for a script that did not run through, its error in at most 300 characters; see Outcomes)
`$summary_table`	per-paper aggregates (`repro_code_n`, `repro_runnable`, `repro_missing_inputs`, `repro_deps`, `repro_ran_ok`, `repro_tests_reported`, `repro_tests_matched`; the first five add up the R and the Python counts), and, when executed, `repro_scripts_n` (the scripts with a result of their own), `repro_not_analysis_n` (files of software that were not run), `repro_not_downloaded_n`, and one `repro_fail_<reason>` column per reason (see Outcomes)
`$run_results`	per-script execution record: `outcome`, `reason`, `error`, `error_type`, `undefined_var`, captured `stdout` / `stderr`, `elapsed` time, `script_lines`, `captures` (the raw captured R objects; always empty for Python)
`$install_results`	per-dependency install record: `package`, `source`, `installed`, `message`, `via_archive`, `category` (for Python, always `FALSE` and `NA`)
`$recorded_versions`	the recorded package versions the run used (see Recorded package versions); `NULL` when none were used
`$stat_output`	per-file extracted statistical output: the structured JSON document and the flattened result rows
`$match_table`	the full `match_reported_output()` result, when there was output to match against
`$modifications`	per-file modification detail: `file_name`, `change_type`, `detail` — path rewrites (`path_rewrite`) and names spelled otherwise than in the deposit (`name_corrected`), listed even when `execute = FALSE`, and, when executed, among others: `setwd()` or `os.chdir()` removal (`setwd_removed`), font substitution (`font_replaced`, R only), added `library()` calls (`library_injected`, R only), a script loaded first (`loaded_first`), `groundhog`, `checkpoint` or `renv` calls replaced (`env_call_replaced`), Windows paths given `/` (`windows_path`), R’s old defaults (`old_default`), Stan code given the syntax of Stan 2.33 and later (`stan_syntax`), a deposit copy used for a Git LFS pointer (`lfs_pointer`), and the later runs of Running scripts again (`run_again`, `output_copied`, `file_placed`). Empty when nothing needed modifying
`$report`	formatted report sections, including per-script output and the self-reproducible JASP/jamovi results; the R sections come first, then a “Python code” section when there is Python code. When a paper links several repositories, a line per repository gives its scripts and how many ran to the end, so a reader can see when failures lie in software by others
`$summary_text`	plain-text bulleted summary, as one text also when the paper has R and Python code
`$checklist_section`	(only when `execute = TRUE`) the section the checklist report (`report_type = "checklist"`) shows for this check: how the scripts were run, what happened to each script in run order, other problems in the files, how many of the paper’s reported tests were found in the output of the code, and, when scripts did not run to the end, guidance under “How to make your code run on another computer” (see below)
`$brief`	`TRUE` when code was run and at least one script ran to the end, stopped with an error, or ran out of time; the brief report then keeps this check whatever its light
`$sandbox`	(only present when `keep_sandbox = TRUE`) the path to the materialised throwaway directory — see below

28.17.1 Guidance for the authors

When scripts did not run to the end, the checklist section guides the authors in making their code run, under the heading “How to make your code run on another computer”. Each problem gets a block: what is wrong, the scripts it affects with the file, package or object involved, why the script fails, and the steps to fix it, written so that a student can follow them. The blocks come in the order in which to fix them, because a problem near the top can cause the ones below it: a file another script should have made, files missing from the deposit, a location that exists only on the authors’ computer, a renamed file, a file only inside a zip file, code written for older versions of its packages (with how to record the versions with renv or pip freeze), a package that could not be installed (with the successor of a package removed from CRAN, such as sf or terra for rgdal), a program or other software the code needs, an API key, a Microsoft Access database, something the script does not create or load itself, a syntax error, the time and memory limits, downloads, and a person’s choice. Where the output does not show the cause, the block says so.

28.18 Options

Argument	Default	Effect
`execute`	`FALSE`	run the paper’s code; off by default
`sandbox`	`"process"`	`"process"` runs each R script in an isolated `callr` subprocess and each Python script in a `python` subprocess (isolates a crash only); `"docker"` runs inside a locked-down container (isolates filesystem and network too)
`docker_use_declared_version`	`FALSE`	when `sandbox = "docker"`, use the paper’s own declared R version (slower, from-scratch install) instead of the fast pre-built `metacheck_r_large` image; R code only
`install_missing`	`NULL`	`TRUE` installs declared dependencies into a throwaway library before running (Python packages with `pip`, see Running the code); `FALSE` installs nothing; `NULL` installs nothing, except the R packages the scripts load when `recorded_versions = TRUE` and the repository records their versions
`recorded_versions`	`FALSE`	install the R packages the scripts load at the versions the repository records — see Recorded package versions
`install_timeout`	`NULL`	time in seconds for installing a paper’s R and Python packages; `NULL` uses `timeout`, or an hour when recorded versions are installed. With a clock time as `total_timeout`, installing takes at most half of the time that is left
`cran_install_main`	`FALSE`	(only for R code, under `sandbox = "process"`) install CRAN-source dependencies into your real default library instead of a throwaway one, so they persist across papers in a batch run; ignored under `sandbox = "docker"`, where a container never has access to your host library. A recorded version always goes into the throwaway library
`timeout`	`600`	per-script timeout in seconds
`total_timeout`	`3600`	time for all scripts together, in seconds from when the first script starts, or a clock time (`POSIXct`); `NULL` or `Inf` for no limit — see Time limits
`internet`	`FALSE`	(only under `sandbox = "docker"`) let the scripts use the internet while they run, within the download limits; the default can be set with `options(metacheck.repro_internet = TRUE)` — see Scripts that download their data
`keep_sandbox`	`FALSE`	keep the throwaway run directory and return its path as `$sandbox`, to inspect exactly what ran
`workers`	`1`	when `paper` is a paperlist and `sandbox = "docker"`, run this many papers’ checks concurrently — see Batch mode
`results_dir`	`NULL`	with `workers > 1`, write each paper’s result to `<paper_id>.rds` in this directory as soon as that paper finishes
`local_path`, `local_only`		`local_path` adds the files in a local folder; `local_only = TRUE` skips the online repository lookups. With a paper, `local_path` alone does not replace the repository the paper links to: that repository is still downloaded, and its scripts run alongside the local ones, so a local and an online version of the same script can both run. Give both to run only the local files
`model`, `params`		passed on to the internal `data_check` run; since this module runs `data_check` with the use of a language model turned off, they have no effect at present
`cache`	`FALSE`	forwarded to the internal `data_check`/`code_check` runs: keep downloaded files in the persistent on-disk cache and reuse them on later runs
`download`, `skip_types`, `peek_zips`, `max_file_size`, `max_download_size`	(same as `data_check`’s own defaults)	forwarded to the internal `data_check` run — see the Data Check chapter for what each one controls
`skip_on_api_limit`	`FALSE`	give up immediately on a confirmed API rate limit instead of waiting, forwarded to the internal repository/data/code checks
`tables_dir`	`NULL`	a directory of per-paper `.rds` files from a prior full corpus build (written by the internal `capture_module_tables()`); when set, a reproducibility-only re-run reads that paper’s already-computed `data_check`/`code_check` results from disk instead of re-running them

Warning

These arguments only take effect when data_check has not already been run for this paper. If you ran data_check yourself first, with other arguments, the module uses that run’s files as they are. When you ran it yourself with llm_use(TRUE), download, skip_types, and peek_zips are also part of the key the LLM response cache uses for data_check’s column classification — changing any of them between runs on the same paper produces a different file set and therefore a cache miss, even though the paper and the underlying question are identical.

keep_sandbox = TRUE is worth using deliberately rather than as a debugging afterthought: the directory it preserves contains the files the code ran against (a copy of the repository, with the rewritten scripts), output/ (anything a script’s write calls were redirected into), and — whenever there is any extracted statistical output at all, whether from execution or from a self-contained JASP/jamovi/.spv/.smcl/.out file — a statistical_output/ folder with the structured results documents. convert_psychds(), which is meant to put this folder into a Psych-DS archive, is not yet available in the released version or in dev.

28.19 Notes and limitations

Honest limits worth keeping in mind:

The static phase cannot always tell that a file a script reads is an intermediate produced by another script under a different name, so it may report it as an “absent” input.
Helper scripts source()d through a dynamically built path do not get a source edge, so they order arbitrarily with basis none.
Only R and Python are executed. SPSS syntax, Stata syntax, SAS, and MATLAB are reported but not run; SPSS/Stata/JASP/jamovi/Mplus results are read from their own output files, and a notebook’s saved results from the notebook, rather than recomputed.
Each Python script runs in its own process, so a script written to run after another one in the same Python session fails on the names that other script defines; there is no corrective re-run for a NameError in Python. Python dependencies come only from import lines, so a package imported under a name other than its pip name is installed correctly only when that name is in the module’s built-in list.
The output parser handles the common R output shapes (one-line tests like t.test; fixed-width tables like summary(lm) and aov), printed Python results such as scipy’s, and JASP/jamovi/SPSS-Viewer/Mplus/notebook result tables. Unusual or bespoke output may parse incompletely; statistics with no STATO class are kept under their own header rather than semantically typed.
Extracted values inherit the display precision the software reported (p = "< .001" is a threshold, t = 2.77 is rounded), so matching is precision-aware rather than exact — it compares what was reported, which is what you want for a reproducibility check.
sandbox = "process" genuinely does run downloaded code on your own machine with your own user’s filesystem and network access. If you are not confident in a paper’s code, use sandbox = "docker" instead — the two are not equivalent safety-wise, only equivalent in what they check.
The pre-built R Docker image is built for one specific R version at a time; if a paper’s own declared version matters to you specifically (not just “some recent R”), use docker_use_declared_version = TRUE and accept the slower, from-scratch install.

28.20 Open for feedback

Making this software was a challenge. We have validated the code by running it on more than 3000 scientific papers, with hundreds of folders of data and code, across dozens of data repositories. But there will be bugs. Software this complex, that tries to process the almost infinite ways in which researchers can share data and code, is bound to run into bugs. When thousands of people try this out, for some it will fail on their very first try (while for many others, it will run fine). If you find a bug, please raise an issue on Metacheck’s GitHub page. You need to share the data or code files that lead to the problem with us, or we can’t fix the problem. If you can’t share the data and code, you can ask an AI agent to analyze the problem, which might be enough to solve it as well. If the tool lacks features you would like to have, let us know, and we will do our best to build them in.

28.1 Computationally reproducing a simple Stroop experiment

28.2 Running the code locally, or using our server on SURF

28.3 Running the check in R

28.4 What it checks

28.5 The static phase, without running code

28.6 Worked example: building a small reproducible fixture

28.7 The static assessments, function by function

28.7.1 Dependencies: repro_dependencies()

28.7.2 Paths: repro_rewrite_paths()

28.7.3 Run order: repro_file_io() and repro_run_order()

28.7.4 Missing inputs: repro_missing_inputs()

28.7.5 Two more static helpers you may not need directly

28.8 Two ways to run the code

28.9 The pre-built Docker image, and why it exists

28.10 Declared R versions, and why they matter for Docker

28.10.1 Recorded package versions

28.11 Checking many papers at once: batch mode

28.12 Running the code: what happens during execution

28.12.1 Where the code runs

28.12.2 Time limits

28.12.3 Scripts that download their data

28.12.4 Outcomes

28.12.5 Running scripts again

28.13 Python code

28.14 What if there’s no R or Python code at all?

28.15 Extracting the statistical output

28.16 Matching the reported results against the output

28.17 The traffic light and what you get back

28.17.1 Guidance for the authors

28.18 Options

28.19 Notes and limitations

28.20 Open for feedback

28.7.1 Dependencies: `repro_dependencies()`

28.7.2 Paths: `repro_rewrite_paths()`

28.7.3 Run order: `repro_file_io()` and `repro_run_order()`

28.7.4 Missing inputs: `repro_missing_inputs()`