This module is currently under development, and not available in the main version of Metacheck. To try it out, install the development version of Metacheck:
remotes::install_github("scienceverse/metacheck@data_check")Where code_check asks “is this code written well?”, reproducibility_check asks a harder question: could this code actually be run on this data — and do the numbers it produces match the ones the paper reports? It is the module that stands closest to the goal of computational reproducibility.
The module works in two phases, and you choose how far to go:
execute = TRUE) that actually runs the code, either in a subprocess on your own machine or inside a locked-down Docker container, against a throwaway copy of the data, captures the statistical output it produces, and — together with any self-contained JASP/jamovi files in the deposit — lets you check whether the reported results are reproduced.Executing downloaded code runs it — somewhere. It is off by default and is a deliberate, opt-in action (execute = TRUE). How much risk that carries depends entirely on sandbox, a second argument covered in detail in Two ways to run the code below: the default (sandbox = "process") only isolates a crash, not the filesystem or network, so code could still delete files or reach the internet; sandbox = "docker" is a real containment boundary (no network, read-only filesystem, non-root user) and is what you should reach for when running code you do not personally trust.
This module builds on code_check, psychds_check, and data_check, and inherits their live network calls to retrieve the repository files. You need an internet connection to run the code below.
The default is the static phase — it does not run anything:
paper <- demopaper()
mo <- module_run(paper, "reproducibility_check")
mo$traffic_light#> [1] "yellow"cat(mo$summary_text)#>
#> - We assessed 1 R code file for reproducibility (static analysis; no code was run).
#> - 0 files appear runnable so far (parses, inputs resolve, placeable in the run order).
#> - 1 referenced input unavailable (0 withheld due to size).
#> - 1 installable dependency detected.The table has one row per R code file, carrying the static assessments as columns:
| file_name | parses | run_order | order_basis | paths_rewritten | runnable | |
|---|---|---|---|---|---|---|
| Code/Study 1.r | Code/Study 1.r | TRUE | 1 | numeric | 0 | FALSE |
A file is marked runnable only when it parses, all of its referenced inputs resolve (they are present, or produced by an earlier script), and it can be placed in the run order. That is a deliberately strict bar for a static check: runnable = FALSE does not mean the code is broken, it means the module could not confirm, without running it, that everything the script needs is in place. When it is FALSE, the column not_runnable_reason names exactly why — one of parse_error, unplaceable, or missing_input, checked in that priority order — and unresolved_inputs names the specific file(s) that could not be resolved.
Two further list-columns, reads and writes, carry the basenames each script reads and writes (as repro_file_io() extracted them) directly on the table — useful if you want the code↔︎data provenance for a file without recomputing it yourself:
mo$table$reads#> NULLmo$table$writes#> NULLTo see every static assessment concretely, it helps to build a tiny, self-contained example rather than rely on whatever a real deposit happens to contain. The example below creates two scripts — one that writes an intermediate file, one that reads it back — and a plan that tells the module where a Psych-DS release would put the data.
tmp <- withr::local_tempdir()
writeLines(c(
'dat <- read.csv("data.csv")',
'summ <- data.frame(mean_x = mean(dat$x))',
'write.csv(summ, "intermediate.csv", row.names = FALSE)'
), file.path(tmp, "0_prep.R"))
writeLines(c(
'summ <- read.csv("intermediate.csv")',
'print(summ$mean_x)'
), file.path(tmp, "1_analysis.R"))
writeLines("x,g\n1,a\n2,a\n3,b\n4,b", file.path(tmp, "data.csv"))The rest of the chapter uses these two files, 0_prep.R and 1_analysis.R, to demonstrate each static assessment on its own terms.
Each static assessment is an exported helper, so you can run it on your own code independently of the module.
repro_dependencies()
repro_dependencies() walks the code and returns the distinct packages it loads, tagged with where each would be installed from — a distinction code_check’s package list does not make. A library() call resolves to CRAN; a remotes::install_github("user/repo") call is read from the source the code names (with the pinned @ref kept when present); and packages that ship with R are tagged base, because a run never needs to install them.
code_text <- c(
"library(dplyr)",
"remotes::install_github('tidyverse/ggplot2@v3.4.0')",
"x <- stats::sd(1:10)"
)
repro_dependencies(code_text)#> package source ref base
#> 1 dplyr cran <NA> FALSE
#> 2 remotes cran <NA> FALSE
#> 3 stats base <NA> TRUENote that a namespace-qualified call — stats::sd(...) above — is picked up as a dependency even without an explicit library() call; the third row shows stats tagged base. repro_dependencies() also accepts a list of per-file character vectors and pools them into one result: when the same package is named both bare (library(pkg)) and via an explicit GitHub source somewhere else in the pooled files, the GitHub row wins, since it carries the real install source.
The dependency list is deliberately name-only, not version-pinned: static analysis cannot recover which version of a package was used. This is not a limitation to apologise for — it is the point. A reproduction runs against the current versions of these packages, and if it breaks under current versions, that break is itself a finding: it argues that the authors should have pinned their versions.
That said, reproducibility_check does not have to guess blindly at what “current” means. As of the version documented in this chapter, code_check itself checks whether a deposit declared its environment at all — see Declared R versions below, which explains how this feeds directly into the Docker execution path.
repro_rewrite_paths()
A script reads and writes data by relative path — read_csv("raw/x.csv"), saveRDS(m, "../out/model.rds"). When the release is re-laid-out into the Psych-DS structure, with data under data/ and per-study study-<group>/data/ folders, those paths no longer resolve. repro_rewrite_paths() maps each referenced path to its new location, using the file plan that psychds_check produces.
Matching is by basename: a script’s ../data/x.csv and the repository’s raw/x.csv are the same file seen from different working directories, so the prefix is ignored. When several files share a basename — a demographics.csv in more than one study — the ambiguity is resolved by study group: the candidate whose target path is in the same study as the script wins. Only when that still leaves more than one candidate is the reference left unrewritten and flagged as ambiguous, rather than guessed.
plan <- data.frame(
file_name = c("demographics.csv", "scores.csv"),
target_path = c("study-ex1/data/source-demographics_data.csv",
"study-ex1/data/source-scores_data.csv"),
current_path = c("ex1/demographics.csv", "ex1/scores.csv")
)
repro_rewrite_paths('d <- read.csv("data/demographics.csv")',
"ex1/analysis.R", plan)#> ref basename matched
#> 1 data/demographics.csv demographics.csv TRUE
#> target ambiguous n_candidates
#> 1 study-ex1/data/source-demographics_data.csv FALSE 1Two refinements worth knowing about if you inspect this output closely:
repro_rewrite_paths() recognises this rather than flagging it.structure_df (the data_check structure table). A paper linking an OSF component and its own “Archive of OSF Storage” snapshot can have the same physical file grouped into two different (and mostly wrong) study labels by upstream heuristics; repro_rewrite_paths() hashes each candidate’s actual bytes first, so byte-identical mirrors count as one candidate rather than several conflicting ones.repro_file_io() and repro_run_order()
Scripts in a repository usually depend on each other, and running them in the wrong order fails for reasons that have nothing to do with reproducibility. repro_file_io() first extracts, for every file, the data it reads, the data it writes, and the scripts it sources:
#> file_name reads writes sources
#> 1 0_prep.R data.csv
#> 2 1_analysis.R intermediate.csvrepro_run_order() then derives an order from three signals, in decreasing strength:
intermediate.csv and another reads it, the writer must run first (a hard data dependency);source() edges — if A source()s B, B must run first;0_prep.R before 1_analysis.R, and also embedded numbering such as Exp1_02_... before Exp2_1_... (a weak tie-breaker, not a dependency). The numeric key is the vector of every digit run anywhere in the filename, not just a leading prefix, so Exp1_02_preprocessing.R correctly sorts before Exp2_1_import.R.The scripts are then topologically sorted. A dependency cycle (A reads a file B writes, and vice versa) cannot be ordered and is reported; so is an ambiguous ordering, where nothing — no dependency and no numbering — distinguishes the order of two or more files.
repro_run_order(io)#> file_name order depends_on order_basis
#> 1 0_prep.R 1 numeric
#> 2 1_analysis.R 2 numeric0_prep.R runs first even though its own name sorts it that way anyway — but critically, the order_basis column reads "dependency", not "numeric", because the real reason is the data dependency: 1_analysis.R reads intermediate.csv, which 0_prep.R writes. order_basis is recorded on the file that has the incoming edge (the dependent file), not the file it depends on — so it is 1_analysis.R’s row that carries "dependency" and names 0_prep.R in depends_on.
repro_missing_inputs()
When a script reads a file that is not present, the reason matters. repro_missing_inputs() cross-references each unresolved read against the file plan and the download records, and classifies it as one of:
withheld_size — the file is in the repository, but was not downloaded because it exceeded a size cap. This is a size-cap issue, not a reproducibility failure: raising max_file_size / max_download_size (in data_check) would fetch it.in_repo_not_downloaded — listed in the repository but not present locally for another reason.absent — not in the repository at all. The script reads a file the authors did not share (or that an earlier, un-captured step was meant to produce).This distinction is the reason the static phase is worth running before a manual reproduction attempt: a run that fails because a 3 GB data file was skipped by the size gate is a completely different situation from a run that fails because the input data were never shared, and only the latter is a genuine openness problem.
Two further exported helpers feed the module internally and are rarely called on their own, but are worth knowing about if you are extending the checks:
repro_defined_vars(code_text_list) returns, for each file, the variable names it assigns at its top level (not inside a function). This is what powers the corrective re-run described below: when a script fails with object 'x' not found, the module looks for exactly one other file whose repro_defined_vars() result includes x, and — if there is exactly one — infers that it must run first.repro_materialize_layout() and repro_write_scripts() build the throwaway Psych-DS-shaped directory the execution phase actually runs against, and write the path-rewritten scripts into it. You will see their effects described below (the data/, output/, and statistical_output/ folders), but you would only call them directly if you were building your own execution pipeline on top of metacheck’s static analysis.Set execute = TRUE to actually run the paper’s code. A second argument, sandbox, decides where:
# on this machine, in an isolated subprocess (the default)
mo <- module_run(paper, "reproducibility_check",
execute = TRUE, sandbox = "process",
install_missing = TRUE, timeout = 600)
# inside a locked-down Docker container
mo <- module_run(paper, "reproducibility_check",
execute = TRUE, sandbox = "docker",
install_missing = TRUE, timeout = 600)The two options carry genuinely different risk, and the difference is worth understanding precisely rather than taking on faith:
sandbox = "process" (default) |
sandbox = "docker" |
|
|---|---|---|
| Mechanism | a callr subprocess on your machine |
a docker run container |
| Isolates a crash? | Yes | Yes |
| Isolates the filesystem? | No — the code can read, write, or delete anywhere your R session can |
Yes — --read-only, with only the throwaway sandbox directory writable |
| Isolates the network? | No — the code can reach the internet freely |
Yes — --network none during the run phase |
| Runs as | your own user account | a fixed non-root user (uid 1000) inside the container |
| Requires | the callr package |
Docker installed and running, plus the processx package |
| Speed | fast — no container overhead | pre-built image (see below) makes this close to the process backend for most papers |
sandbox = "process" isolating only a crash is worth restating plainly: callr runs the script in a separate R process so that a segfault or a fatal error does not take down your own R session, but that subprocess is still the same operating-system user, the same filesystem, and the same network as the R session that launched it. Nothing stops a malicious or merely careless script from calling unlink() on files far outside the sandbox, or system()-ing out to install something, or exfiltrating data over the network. If you are running code from a repository you have not personally reviewed, sandbox = "docker" is the option that actually delivers on the word “sandbox”.
sandbox = "docker" achieves real containment with these docker run flags, applied to every script individually during the run phase:
--network none --read-only --tmpfs /tmp
--user 1000:1000 --cap-drop ALL
--security-opt no-new-privileges --pids-limit 512
-v <sandbox>:/sandbox -v <library>:/rlib:roNetwork is fully disabled, the filesystem is read-only outside the mounted sandbox (with only /tmp and the sandbox itself writable), the container runs as a fixed non-root user, all Linux capabilities are dropped, privilege escalation is blocked, and the number of processes the container can spawn is capped. Before checking any of this, call repro_docker_available() to confirm Docker is installed and its daemon is actually running — reproducibility_check() calls this for you and stops with a clear message if it is not, rather than failing partway through a batch of papers with a cryptic error.
repro_docker_available()
#> $ok
#> [1] TRUE
#> $msg
#> [1] ""The dependency-install phase under sandbox = "docker" runs with the network on (it has to, to reach CRAN or GitHub) but is otherwise hardened the same way — non-root, capabilities dropped, no new privileges. Only the per-script run phase disables the network entirely, since that is the phase running the paper’s own, potentially untrusted, analysis code.
The first time you think through what sandbox = "docker" needs to do to run a real paper’s code, an obvious problem appears: most papers’ scripts library() several packages that are not part of base R, and a container starts with nothing installed. Naively, every single reproducibility_check() call would need to install a paper’s entire dependency list from scratch, inside a freshly-started container, before it could run a single script — for R packages with compiled C++ code (rstan, lme4, and dozens of others), that can mean minutes of compilation per paper.
To avoid that, Docker containers under sandbox = "docker" default to a pre-built image, ghcr.io/scienceverse/metacheck_r:latest, rather than a bare R installation. That image already has R, Quarto, and the roughly 650 most common packages a real corpus of psychology-science papers’ code actually uses — found by scanning thousands of real .R/.Rmd/.qmd files for their library()/require() calls and pre-installing everything that showed up often enough to be worth the build cost. When a paper’s dependencies are already in that list — which, for an ordinary tidyverse-and-lme4-shaped analysis, they usually are — the install phase does almost nothing, because requireNamespace() already finds the package. Only genuinely uncommon or paper-specific dependencies still need a real, from-source install.
The image, its Dockerfile, the full package list, and detailed notes on how it was built (including some genuinely non-obvious bugs found along the way — a package can report a successful install and still fail to load, because a runtime shared library it needs is missing from a minimal image) are all in a dedicated repository: scienceverse/metacheck_docker_reproducibility. If you want to build your own variant — a smaller image with only the most common packages, or one that adds LaTeX for PDF rendering, or CmdStan for cmdstanr-based Bayesian models — that repository is the starting point, and its README documents exactly which levers actually reduce image size and which ones (surprisingly) do not.
The pre-built image is fast, but it was built against one specific R version. A paper that deposited an renv.lock file, or a sessionInfo() text dump, or that pins its environment with groundhog/checkpoint, has told you exactly which R version its authors actually used — and that might not be the version the pre-built image happens to run.
code_check is where this gets detected, as part of its own “Reproducible Environment” check: it looks for an renv.lock file anywhere in the repository (parsing its declared R version and every locked package + version), a sessionInfo()/session_info() dump (matched by filename — sessionInfo.txt, session_info.txt, and similar, or embedded in a README), or an actual groundhog::groundhog.library()/checkpoint::checkpoint() date-pin call in the code — a bare library(groundhog) with no pinning call does not count, since loading the package is not evidence anything was actually pinned.
cc <- module_run(paper, "code_check")
cc$traffic_light#> [1] "yellow"When nothing was declared, code_check’s own report explicitly says so and recommends depositing one — this is itself a piece of reproducibility hygiene worth checking regardless of whether you ever run the code.
reproducibility_check() has a parameter, docker_use_declared_version, that decides what to do with this information:
# default: always use the fast pre-built image, regardless of what the
# paper declared
mo <- module_run(paper, "reproducibility_check",
execute = TRUE, sandbox = "docker")
# opt in to matching the paper's own declared R version exactly
mo <- module_run(paper, "reproducibility_check",
execute = TRUE, sandbox = "docker",
docker_use_declared_version = TRUE)By default (docker_use_declared_version = FALSE), the module always uses the pre-built metacheck_r image, prioritising speed. If the paper did declare a version, you get a warning naming the mismatch and explaining how to opt out of it — this is a deliberate design choice: silently running a paper’s code against a different R version than the one its authors used could, in principle, change results, and that substitution should never happen invisibly.
Warning message:
reproducibility_check(sandbox = "docker"): this paper declared R version
4.3.1, but docker_use_declared_version = FALSE (the default), so the run
uses the pre-built metacheck_r image's own R version instead, for speed.
Set docker_use_declared_version = TRUE to match the paper's declared
version exactly -- slower, since every dependency then installs from
scratch rather than using the pre-built image.Setting docker_use_declared_version = TRUE switches to a bare rocker/r-ver:<declared version> image with nothing pre-installed — genuinely slower, since every dependency now installs from source, but it matches the paper’s own environment exactly. When a paper declared no version at all, this option falls back to bare rocker/r-ver:latest rather than silently reusing the pre-built image either — if you explicitly asked for version-matching, getting the fast pre-built image back through the back door when there was nothing to match against would defeat the point of asking.
Whichever sandbox you choose, the mechanics are the same:
install_missing = TRUE), never your main R library — under sandbox = "docker" this is always into a throwaway library inside the container, since a container has no access to your host library at all;timeout, capturing the outcome and the console output.mo <- module_run(paper, "reproducibility_check",
execute = TRUE, install_missing = TRUE,
timeout = 600, keep_sandbox = TRUE)Each script gets one of six outcomes, added to the $table in an outcome column:
| Outcome | Meaning |
|---|---|
ran_ok |
the script ran to completion without error |
errored |
the script raised an error (captured, with the message and output) |
timed_out |
still running at the timeout cutoff — not a failure, just slow; raise timeout for legitimately long analyses (e.g. Bayesian sampling) |
skipped_missing_inputs |
not run, because an input it needs is unavailable |
not_parsed |
not run, because the script does not parse |
dependency_unavailable |
not run (or failed immediately), because the script’s own package could not be installed from live CRAN, the CRAN Archive, or a named GitHub/URL source — an infrastructure limitation, not a defect in the paper’s code |
dependency_unavailable is deliberately excluded from forcing the traffic light red, unlike errored/timed_out — a package that has genuinely vanished from every source metacheck tries is not evidence the paper’s own analysis is broken.
An error whose message is object '...' not found is called out specially: it usually means the script expects a variable another script defines, so it was meant to be source()d into a larger session rather than run alone. When that missing variable is defined by another script in the repository — checked via repro_defined_vars() — the module infers the correct order (adding a “definer runs first” edge and re-sorting), re-runs once, and reports the corrected outcome. It does not iterate further: a second object not found error after that one corrective pass is accepted as the final result rather than chased indefinitely.
Two habits in shared code are handled automatically. A setwd() call — which hardcodes where the code must run and would override the sandbox — is commented out before the script runs, and the report notes that the script is “reproducible after ignoring a setwd() that should not have been in the code.” An SPSS data file (.sav/.zsav/.por) with no accompanying syntax (.sps) turns the light red, because that analysis cannot be reproduced from the deposit; the same is true for a Stata .dta file with no .do. The report recommends depositing analyses as jamovi or JASP files instead, since those bundle data and analyses together and need no code to be re-run at all.
Byte-identical duplicate files are only run once. A paper that links several repository components mirroring the same materials — a “live” OSF component, an “Archive of OSF Storage” snapshot, a view-only anonymised link — would otherwise have code_check list the same script once per mirror, and the module would run it two, four, or more times for no reason. reproducibility_check hashes each resolved script’s actual content and drops exact duplicates before running, naming which kept file each dropped one mirrors in the report.
Not every deposit has R code to run, and the module handles several of those cases explicitly rather than falling through to a generic “nothing to check” result:
.jasp) or jamovi (.omv) file, with no R code anywhere — these formats bundle the data and the analyses and the rendered results together, so they are reproducible from the file itself. The module still extracts and matches their statistics (see below); the traffic light is yellow, not na, because there is a real, substantive finding to report..do files with a .smcl output log, but no separate .dta — the .smcl log both recovers the executed syntax (as a synthetic .do file code_check can check) and carries the results it printed, extracted the same way as JASP/jamovi output."Stata code without data or output", with the light set to info, because this is real, actionable guidance (deposit a .smcl log) rather than an empty check.code_check recognises but this module does not execute: each gets its own named block explaining specifically what is missing, rather than one generic message.na.# a paper whose only "code" is a jamovi file
mo <- module_run(jamovi_only_paper, "reproducibility_check")
mo$traffic_light
#> [1] "yellow"
mo$summary_text
#> - We assessed 1 self-reproducible output file (JASP/jamovi/SPSS-Viewer/Mplus) instead of R code.
#> - 47 statistics stored from the extracted output.Running the code is only half of reproducibility — the other half is the numbers it produces. Several kinds of deposit already contain, or can be made to produce, a paper’s statistical results, and the module extracts all of them into one common, structured form via read_stat_tables():
.jasp) and jamovi (.omv) files are self-contained: they bundle the data and the analyses and the rendered results. These are reproducible from the file itself — no code needs to run..spv) likewise carry their own results structure, and — since the .spv’s own structure XML records the exact SPSS syntax that produced each table — the module also recovers that syntax as a synthetic .sps file, checked by code_check like any other syntax file..smcl) are Stata’s own rendered output, echoing every command that was run; the module both extracts the printed results and recovers the commands as a synthetic .do file..out) always self-documents: its own “INPUT INSTRUCTIONS” section holds the exact syntax that produced it, so there is no “data without syntax” failure mode for Mplus the way there is for SPSS and Stata..ipynb) save each code cell’s output into the file, so a Python analysis’s printed statistics are recoverable with no Python installed and nothing re-run.summary(lm(...)), t.test(...), an anova table). The module parses that captured output into the same structured form, and — where possible — also captures the actual R objects a script’s statements return (an htest, a summary.lm), not just their printed text, so a statistic’s exact value and identity survive even when the console text alone would be ambiguous (the same "W" is Shapiro–Wilk’s after shapiro.test() but the rank sum after wilcox.test()).Every extracted statistic is typed with the STATO ontology (the Statistical Methods Ontology), so a t-value carries the accession for “t-statistic”, a p-value the accession for “p-value”, and so on; a statistic with no STATO class keeps its own column header as a plain label, so nothing is dropped.
stat_results_long()’s full output has these columns: paper_id, source_file, test_id, result_id, analysis, table_title, row_label, statistic, stato_label, stato_iri, value, and model_ref (the last linking a captured statistic back to the fitted-model object it came from, when one exists — the same mechanism that makes anova(m) and summary(m)$coefficients findable as describing the same underlying model even though neither call names the other).
The full set of results for a paper is also written to a structured JSON document via stat_output_json() — one per source file, with a schema, schema_version, paper_id, source_file, source_format, and an analyses array of typed results. This is worth being precise about: it is a metacheck-native schema, not ISA-JSON. An earlier version of the module modelled this document on the ISA (Investigation/Study/Assay) vocabulary from the life sciences, but that forced statistical results into containers (“a t-test result as a Material”) the vocabulary was never designed to hold — a repurposing metacheck itself invented, not an established convention. The current schema drops that borrowed vocabulary entirely in favour of a flat, self-describing document with a small set of named sections, in the spirit of formats like Psych-DS rather than conforming to an external schema that never quite fit.
The final question — the one reproducibility is really about — is whether the results the paper reports appear in the results the analysis produces. match_reported_output() answers it.
A reported statistical test is not a single number; it is a multi-component statement, for example “M = 1.93, SD = 0.76, W = 183.5, p = .791, rb = -0.16”. Metacheck’s text extractor records each of those numbers in the paper’s $eq table, grouped so that the numbers from one reported test stay together. match_reported_output() recomposes each reported test from its components, then checks whether those component values co-occur in a single analysis of the extracted output.
Matching whole tests, rather than lone numbers, is what makes the check trustworthy: any single value will coincidentally appear somewhere in a large enough set of output numbers, but several components of one test agreeing at once is essentially never accidental.
m <- match_reported_output(paper, mo$stat_output)
attr(m, "summary")
#> $n_tests 19
#> $n_found 14
#> $n_full 11 # every component of the test co-occurs in one analysis
#> $n_partial 3 # most components matched
#> $n_missing 5Each row of the result is one recomposed reported test, with where it was found. The full column set is text_id, grp_id, reported, n_components, n_matched, found (logical), match_values, not_matched, source_file, analysis, confidence ("full", "partial", or "none"), and plausible_split:
m[m$found, c("reported", "n_matched", "confidence", "source_file", "analysis")]
#> reported n_matched confidence source_file analysis
#> M=1.93 SD=0.76 W=183.5 ... 5 full experiment_1.omv One Sample T-Test
#> p=0.001 d=0.46 2 full across_all.omv Paired Samples ...So a reported test is not merely flagged as “found” — the module tells you which deposited file and which analysis produced the matching numbers, which is exactly the provenance a reader needs to trust that the paper’s results are reproducible.
By default, min_components = 1, so even a lone reported statistic — a bare correlation coefficient with no accompanying test statistic — is checked. This is a change from an earlier version of the module, which defaulted to requiring at least two components before treating something as a matchable “test” at all; if you want that stricter behaviour, set min_components = 2 explicitly.
plausible_split deserves a word of explanation, since it is easy to misread. Some reported values only match after being separated from the other values they were originally reported alongside — a mean split from its own confidence interval, say. plausible_split marks whether that separation is well-supported: TRUE when the split pieces still trace back to the same underlying variable via the output’s own row labels, FALSE when no such link was found. A FALSE here is not necessarily wrong, but it is worth checking manually against the source_file/analysis columns before treating it as a confirmed match; NA means the value was never split in the first place, so the question does not apply.
If your extracted paper has any results tables (not just narrative text), pass include_tables = TRUE to also check statistics reported only in a table’s cells against the extracted output.
| Light | Meaning |
|---|---|
| green | every script parses, its inputs resolve, the run order is unambiguous, no path rewrite is ambiguous, and (when executed) nothing errored |
| yellow | some obstacle short of a full reproduction failure — a parse error, an unresolved input, an ambiguous path or order, or a self-reproducible output file present with no R code to run |
| red | a reproduction failure: code that errored or timed out when run, missing inputs that are entirely explained by size withholding, or SPSS/Stata data deposited without its syntax |
| info | no R code and no self-contained output, but the module found something specific and actionable to say (e.g. Stata syntax with neither data nor an output log) |
| na | genuinely nothing to check — no R code, no self-contained output, no other-language code either |
The static-only red condition is worth stating precisely, since it is easy to over-read: it fires when there are missing inputs and every single one of them is classified withheld_size — not merely when some inputs are size-withheld. Once code is actually executed (execute = TRUE), an execution error or timeout overrides the static signal and forces red regardless — a script that crashes on its own analysis is a genuine reproduction failure, whatever the static assessment said.
| Element | What it contains |
|---|---|
$table |
one row per R file: parses, run_order, order_basis, depends_on, paths_rewritten, paths_ambiguous, setwd_removed, runnable, not_runnable_reason, unresolved_inputs, reads, writes, and (when executed) outcome, error_type, undefined_var
|
$summary_table |
per-paper aggregates (repro_code_n, repro_runnable, repro_missing_inputs, repro_deps, repro_ran_ok, repro_tests_reported, repro_tests_matched) |
$run_results |
per-script execution record: outcome, error, error_type, undefined_var, captured stdout / stderr, elapsed time, script_lines, captures (the raw captured R objects) |
$install_results |
per-dependency install record: package, source, installed, message, via_archive
|
$stat_output |
per-file extracted statistical output: the structured JSON document and the flattened result rows |
$match_table |
the full match_reported_output() result, when there was output to match against |
$report |
formatted report sections, including per-script output and the self-reproducible JASP/jamovi results |
$summary_text |
plain-text bulleted summary |
$sandbox |
(only present when keep_sandbox = TRUE) the path to the materialised throwaway directory — see below |
| Argument | Default | Effect |
|---|---|---|
execute |
FALSE |
run the paper’s code; off by default |
sandbox |
"process" |
"process" runs in an isolated callr subprocess (isolates a crash only); "docker" runs inside a locked-down container (isolates filesystem and network too) |
docker_use_declared_version |
FALSE |
when sandbox = "docker", use the paper’s own declared R version (slower, from-scratch install) instead of the fast pre-built metacheck_r image |
install_missing |
FALSE |
install declared dependencies into a throwaway library before running |
cran_install_main |
FALSE |
(only under sandbox = "process") install CRAN-source dependencies into your real default library instead of a throwaway one, so they persist across papers in a batch run; ignored under sandbox = "docker", where a container never has access to your host library |
timeout |
600 |
per-script timeout in seconds |
keep_sandbox |
FALSE |
keep the throwaway run directory and return its path as $sandbox, to inspect exactly what ran |
local_path, local_only
|
run against local files without online repository lookups | |
model, params
|
passed through to the upstream modules’ LLM calls (data_check, psychds_check), only used when llm_use(TRUE)
|
keep_sandbox = TRUE is worth using deliberately rather than as a debugging afterthought: the directory it preserves contains data/ (the materialised Psych-DS layout), output/ (anything a script’s write calls were redirected into), and — whenever there is any extracted statistical output at all, whether from execution or from a self-contained JASP/jamovi/.spv/.smcl/.out file — a statistical_output/ folder with the structured results documents. This is also what convert_psychds() looks for when building an archive: pass keep_sandbox = TRUE if you want its statistical_output/ folder included, since convert_psychds() does not run reproducibility_check itself and has no other way to obtain it.
Honest limits worth keeping in mind:
source()d through a dynamically built path do not get a source edge, so they order arbitrarily with basis none.t.test; fixed-width tables like summary(lm) and aov) and JASP/jamovi/SPSS-Viewer/Mplus/notebook result tables. Unusual or bespoke output may parse incompletely; statistics with no STATO class are kept under their own header rather than semantically typed.p = "< .001" is a threshold, t = 2.77 is rounded), so matching is precision-aware rather than exact — it compares what was reported, which is what you want for a reproducibility check.sandbox = "process" genuinely does run downloaded code on your own machine with your own user’s filesystem and network access. If you are not confident in a paper’s code, use sandbox = "docker" instead — the two are not equivalent safety-wise, only equivalent in what they check.docker_use_declared_version = TRUE and accept the slower, from-scratch install.If you want to extend the checks, or help harden the execution and matching phases, reach out to the Metacheck development team.