When a paper shares its analysis code, that code should ideally run on someone else’s computer, not just the author’s own. code_check looks at the code files a paper links to and checks for common signs that a script will not run elsewhere — things like a missing comment explaining a step, a hardcoded file path from the author’s own computer, or a call to load a package that was never actually loaded.
Here is the module running on a demo paper (this needs an internet connection, because the module has to fetch the code files from the linked repository):
paper <- demopaper()
mo <- module_run(paper, "code_check")
mo$traffic_light#> [1] "yellow"cat(mo$summary_text)#>
#> - We found 32 R, 0 Python, 0 SAS, 0 SPSS, 0 Stata, 0 Mplus, 0 MATLAB, and 0 JASP code files.
#> - All your code files had comments.
#> - Libraries/imports were loaded in multiple places in bad.R, bad.Rmd.
#> - No pinned R/package environment (renv.lock, sessionInfo(), or groundhog/checkpoint) was found. Consider adding a solution to improve computational reproducibility.Running a module always works the same way: module_run(paper, "module_name"). Saving the result to a variable, here called mo, lets you look at specific pieces of it instead of only the printed summary. mo$table is a table — rows and columns, like a spreadsheet — with one row per code file, showing exactly which problems were found in each one:
| file_name | language | parse_error | code_abs_path | percentage_comment | loaded_files_missing |
|---|---|---|---|---|---|
| test_script.R | R | FALSE | 2 | 0.1666667 | 3 |
| bad.R | R | FALSE | 1 | 0.0714286 | 1 |
| bad.Rmd | R | FALSE | 2 | 0.0666667 | 1 |
| 01.R | R | FALSE | 0 | 0.3333333 | 0 |
| 02.R | R | FALSE | 0 | 0.3333333 | 0 |
| 03.R | R | FALSE | 0 | 0.3333333 | 0 |
| 04.R | R | FALSE | 0 | 0.3333333 | 0 |
| 05.R | R | FALSE | 0 | 0.3333333 | 0 |
| 06.R | R | FALSE | 0 | 0.3333333 | 0 |
| 07.R | R | FALSE | 0 | 0.3333333 | 0 |
| 08.R | R | FALSE | 0 | 0.3333333 | 0 |
| 09.R | R | FALSE | 0 | 0.3333333 | 0 |
| 10.R | R | FALSE | 0 | 0.3333333 | 0 |
| 11.R | R | FALSE | 0 | 0.3333333 | 0 |
| 12.R | R | FALSE | 0 | 0.3333333 | 0 |
| 13.R | R | FALSE | 0 | 0.3333333 | 0 |
| 14.R | R | FALSE | 0 | 0.3333333 | 0 |
| 15.R | R | FALSE | 0 | 0.3333333 | 0 |
| 16.R | R | FALSE | 0 | 0.3333333 | 0 |
| 17.R | R | FALSE | 0 | 0.3333333 | 0 |
| 18.R | R | FALSE | 0 | 0.3333333 | 0 |
| 19.R | R | FALSE | 0 | 0.3333333 | 0 |
| 20.R | R | FALSE | 0 | 0.3333333 | 0 |
| 21.R | R | FALSE | 0 | 0.3333333 | 0 |
| 22.R | R | FALSE | 0 | 0.3333333 | 0 |
| 23.R | R | FALSE | 0 | 0.3333333 | 0 |
| 24.R | R | FALSE | 0 | 0.3333333 | 0 |
| 25.R | R | FALSE | 0 | 0.3333333 | 0 |
| good-example.R | R | FALSE | 0 | 0.4000000 | 0 |
| test_script.R | R | FALSE | 2 | 0.1666667 | 3 |
| Study 1.r | R | FALSE | 2 | 0.1666667 | 3 |
| utils.R | R | FALSE | 0 | 0.5372549 | 0 |
A green light means every code file was checked with no issues found, and the repository records which package versions were used. A yellow light does not mean the code is wrong — it means something was found that is usually worth fixing, such as a file path that only exists on the author’s computer (code_abs_path), a file the script tries to read that was not shared (loaded_files_missing), or code with no comments at all. These checks look at the text of the code, not at whether it actually runs, so always read the flagged lines yourself before deciding whether a problem is serious.
The code_check module analyses the code files shared with a paper. Building on the file list from repo_check, it identifies code files, then checks each for signs of reproducibility problems:
setwd() (which hardcodes an assumption about the working directory);install.packages() (which installs software on whoever runs the script, without asking);renv.lock, a sessionInfo() dump, or a groundhog/checkpoint date-pin call), and for Python code whether it lists its packages (such as a requirements.txt), so package versions do not silently drift between the original analysis and any later reproduction attempt.The module reports on potential problems. An absolute path or a missing file is not always an error — but it is usually worth a second look if you want the analysis to be reproducible. The checks are regular-expression based, so they can miss things or produce false positives on unusual scripts; the module was tried out internally on papers published in Psychological Science, but has not been formally validated.
This module makes live network calls to retrieve and inspect code files from the linked repositories. You need an internet connection to run the code below.
code_lang()
The module first labels every file with its language using code_lang(), which is name-based (by extension), except for .ipynb notebooks and .qmd Quarto documents: these declare their language inside the file (the notebook’s kernel, the document’s engine), so once a copy has been downloaded the language is read from the file itself. An R notebook is then checked as R, and a Python .qmd as Python; a notebook that cannot be read is assumed to be Python.
| Language | Extensions | Analysed? |
|---|---|---|
| R |
.R, .Rmd, and .qmd / .ipynb that declare R |
yes (parsed, commented, paths, setwd(), install.packages(), libraries, loaded files) |
| Python |
.py, and .ipynb / .qmd that declare Python |
yes (except parsing) |
| SAS | .sas |
yes (except parsing) |
| SPSS | .sps |
yes (except parsing) |
| Stata |
.do, .ado
|
yes (except parsing) |
| Mplus | .inp |
yes (except parsing) |
| MATLAB | .m |
yes (except parsing) |
| JASP | .jasp |
listed only |
A .ipynb notebook is JSON, not a plain script: its code cells are extracted from the source array of each cell (the same way .Rmd/.qmd are purled to R code below) before any text-based check runs. .py and .m are plain text and are checked on the same terms as SAS/SPSS/Stata: every text-based check applies, but running the code stays out of scope (that is reproducibility_check’s job, and it runs R and Python code, not MATLAB).
Some file types are never code themselves but contain the exact syntax that produced a piece of output, and the module recovers that syntax as a proper sibling code file before checking it: an SPSS .spv (rendered output) has its syntax recovered as a .sps file, a Stata .smcl log has its echoed commands recovered as a .do file, an Mplus .out file has its “INPUT INSTRUCTIONS” section recovered as an .inp file, and an .html file is sniffed to tell a rendered R Markdown/Quarto report (whose R source is recovered) apart from an unrelated web page. A page of an R package’s website made by pkgdown is not taken for a rendered analysis. The recovered code of a file Metacheck downloaded is saved next to the download; for a file in a folder of your own it is saved in R’s temporary folder, which is removed when R closes, so code_check never writes into your folder.
code_lang(c("analysis.R", "clean.do", "model.sas", "survey.sps", "study.jasp"))analysis.R clean.do model.sas survey.sps study.jasp
"R" "Stata" "SAS" "SPSS" "JASP"A .jasp or .omv file bundles a dataset with its analyses, but it is a binary (zip) archive, so none of the text-based checks apply. A .jasp file is counted and listed for completeness but not opened here; a jamovi .omv file is not counted or listed by code_check at all (their data are read by data_check, and their analysis syntax recovered by import_jasp() / import_omv()). Only R-type files are parsed — the other text languages are checked for everything except parse errors.
code_read() and code_extract_r()
Code is read with code_read(), which guesses the file encoding (readr::guess_encoding()), reads the lines, and converts everything to UTF-8, replacing invalid bytes so a mixed-encoding script never crashes the checks. A file that is not text, such as R data saved under a name ending in .R, gives no lines, with an attribute binary saying what it is (for example “compressed with gzip (as R saves data)”), so it is not checked as code. For R Markdown and Quarto files, code_extract_r() first purls the document (knitr::purl) down to just its R code, so prose and chunk options are not mistaken for code. The module recognises an Rmd/qmd file by its extension and routes it through code_extract_r() automatically (a .qmd that declares Python has its Python chunks extracted instead).
file_path <- demofile("qmd")
code_text <- code_extract_r(file_path) # R code only, prose strippedMetacheck’s website runs this module in three ways:
repo_check, code_check, and data_check on it, and can also run the R and Python code (Reproducibility Check).A folder or .zip file you choose is uploaded to the server. To keep it on your own computer, use the Docker installation, which offers the same website. The website runs the modules with their default settings; to change them, run them in R as described in this chapter.
paper <- demopaper()
mo <- module_run(paper, "code_check")
mo$traffic_light#> [1] "yellow"cat(mo$summary_text)#>
#> - We found 32 R, 0 Python, 0 SAS, 0 SPSS, 0 Stata, 0 Mplus, 0 MATLAB, and 0 JASP code files.
#> - All your code files had comments.
#> - Libraries/imports were loaded in multiple places in bad.R, bad.Rmd.
#> - No pinned R/package environment (renv.lock, sessionInfo(), or groundhog/checkpoint) was found. Consider adding a solution to improve computational reproducibility.The table has one row per code file, with the reproducibility checks as columns:
| file_name | language | parse_error | code_abs_path | percentage_comment | loaded_files_missing |
|---|---|---|---|---|---|
| test_script.R | R | FALSE | 2 | 0.1666667 | 3 |
| bad.R | R | FALSE | 1 | 0.0714286 | 1 |
| bad.Rmd | R | FALSE | 2 | 0.0666667 | 1 |
| 01.R | R | FALSE | 0 | 0.3333333 | 0 |
| 02.R | R | FALSE | 0 | 0.3333333 | 0 |
| 03.R | R | FALSE | 0 | 0.3333333 | 0 |
| 04.R | R | FALSE | 0 | 0.3333333 | 0 |
| 05.R | R | FALSE | 0 | 0.3333333 | 0 |
| 06.R | R | FALSE | 0 | 0.3333333 | 0 |
| 07.R | R | FALSE | 0 | 0.3333333 | 0 |
| 08.R | R | FALSE | 0 | 0.3333333 | 0 |
| 09.R | R | FALSE | 0 | 0.3333333 | 0 |
| 10.R | R | FALSE | 0 | 0.3333333 | 0 |
| 11.R | R | FALSE | 0 | 0.3333333 | 0 |
| 12.R | R | FALSE | 0 | 0.3333333 | 0 |
| 13.R | R | FALSE | 0 | 0.3333333 | 0 |
| 14.R | R | FALSE | 0 | 0.3333333 | 0 |
| 15.R | R | FALSE | 0 | 0.3333333 | 0 |
| 16.R | R | FALSE | 0 | 0.3333333 | 0 |
| 17.R | R | FALSE | 0 | 0.3333333 | 0 |
| 18.R | R | FALSE | 0 | 0.3333333 | 0 |
| 19.R | R | FALSE | 0 | 0.3333333 | 0 |
| 20.R | R | FALSE | 0 | 0.3333333 | 0 |
| 21.R | R | FALSE | 0 | 0.3333333 | 0 |
| 22.R | R | FALSE | 0 | 0.3333333 | 0 |
| 23.R | R | FALSE | 0 | 0.3333333 | 0 |
| 24.R | R | FALSE | 0 | 0.3333333 | 0 |
| 25.R | R | FALSE | 0 | 0.3333333 | 0 |
| good-example.R | R | FALSE | 0 | 0.4000000 | 0 |
| test_script.R | R | FALSE | 2 | 0.1666667 | 3 |
| Study 1.r | R | FALSE | 2 | 0.1666667 | 3 |
| utils.R | R | FALSE | 0 | 0.5372549 | 0 |
Each check is a small exported helper, so you can run it on your own code independently of the module.
code_parse_r()
For R-type files, code_parse_r() attempts to parse() the code and captures any syntax error (rewriting <text> in the message to line for readability). Only R is parsed; other languages report NA (not assessed). A parse error means the file is not valid R and would not run at all.
code_abs_path()
code_abs_path() searches the (comment-free) code for quoted strings that are absolute filesystem paths: a Windows drive path (C:/…, D:\…), a Unix root path (/Users/…), a home path (~/…), or a UNC network path (\\host\share\…). These break on anyone else’s computer. The UNC branch is deliberately strict — it requires the real \\host\share shape — because a bare backslash string is far more often a regex escape ("\\d+") than a network path. The module notes these are potential false positives in code like paste0(dir, "/file.csv").
code_abs_path(c("file <- 'C:/Users/lakens/data.csv'",
"tmp <- '/home/lakens/out.html'",
"rel <- './clean.csv'"))#> # A tibble: 2 × 2
#> abs_path line
#> <chr> <int>
#> 1 C:/Users/lakens/data.csv 1
#> 2 /home/lakens/out.html 2code_setwd()
code_setwd() finds R setwd() calls (on comment-free code). setwd() hardcodes an assumption about the working directory — often an absolute path on the author’s own machine — so the script breaks when run anywhere else. This check is R-only, since setwd() is an R construct.
code_setwd(c("setwd('C:/Users/lakens/project')", "d <- read.csv('x.csv')"))#> # A tibble: 1 × 2
#> setwd_call line
#> <chr> <int>
#> 1 setwd('C:/Users/lakens/project') 1code_install_packages()
code_install_packages() finds R install.packages() calls (on comment-free code, so a commented-out call is never flagged — only a live one). An install.packages() call in analysis code installs software on whoever runs the script, without asking: at minimum disruptive (an unexpected install, network access, a package version the author never tested against), and at worst a security risk if it points at an untrusted repository. Best practice is to document required packages (a README, a renv.lock/DESCRIPTION) rather than install them from inside the analysis code itself. This check is R-only, since install.packages() is an R construct; a namespace-qualified utils::install.packages() call is matched too.
code_install_packages(c("install.packages('ggplot2')", "d <- read.csv('x.csv')"))#> # A tibble: 1 × 2
#> install_packages_call line
#> <chr> <int>
#> 1 install.packages('ggplot2') 1code_library_lines()
code_library_lines() returns the line numbers on which imports/library loads occur, with language-specific patterns (R library/require/renv::install/p_load; SAS %include/libname/filename; SPSS INSERT/BEGIN PROGRAM; Stata do/net install/ssc install). The module computes the maximum gap between successive import lines: if they are more than 3 non-comment lines apart, the file is flagged, because best practice is to load all dependencies in one block near the top so a script’s requirements are easy to see.
code_library_lines(c("library(dplyr)", "x <- 1", "library(tidyr)"), "R")#> # A tibble: 2 × 2
#> code line
#> <chr> <int>
#> 1 library(dplyr) 1
#> 2 library(tidyr) 3code_file_refs()
code_file_refs() extracts the files a script reads or loads — from read/load calls (read.csv, read_csv, fread, readRDS, load, fromJSON, source, and their SAS/SPSS/Stata equivalents like proc import, infile, GET DATA /FILE=, use, import delimited) — by capturing the quoted filename argument, also when it stands on a line after the call, as in read.csv( followed by "./data.csv"). The module compares each referenced file (by basename) against the files actually present in the same repository; anything not found is reported as a missing loaded file, so a reader can check whether the input data were shared. There can be valid reasons a file is not shared (e.g. it holds personal data), which the module cannot evaluate, so it always reports rather than judges.
code_file_refs(c("source('functions.R')", "d <- read.csv('study.csv')"), "R")#> [1] "functions.R" "study.csv"A file the code writes is not a file the repository should hold, so it is not reported as missing. Python’s open() and MATLAB’s fopen() can both read and write: a call whose mode creates the file (Python’s "w", "a" or "x", MATLAB’s "w" or "a") is left out, while a call that reads, or that opens with "r+" (which needs the file to exist already), still counts. In Mplus, the FILE = of the SAVEDATA: command names a file the model writes, and is left out too. Below, only data.csv is a file the script reads:
code_file_refs(c("d = open('data.csv')", "out = open('results.txt', 'w')"), "Python")#> [1] "data.csv"SAS, SPSS and Mplus do not tell capital letters from small ones, and their commands are found however they are written, so PROC IMPORT DATAFILE= and proc import datafile= in SAS, GET FILE= and get file= in SPSS, and FILE = and file = in Mplus are all checked against the repository. Stata does tell them apart, and its commands are found only as Stata writes them.
code_library_names()
code_library_names() extracts the names of the packages/libraries a file loads (R library()/require()/::; Python import/from … import). The module unions and de-duplicates these across all of a paper’s code files (code_packages()), giving the full list of dependencies the analysis relies on — with no version information, since it reads the code’s text and does not look at an installed environment. It returns a table with the package, the kind of call it was found in (source), and the line. Packages loaded through a function of the code’s own, as in lapply(pkgs, function(x) library(x, character.only = TRUE)), are found, and so are the packages of groundhog.library(), with the source value groundhog; a function with another name, such as my.library(), is not read as library().
code_library_names(c("library(dplyr)", "import numpy as np"), "R")#> package source line
#> 1 dplyr library 1.code_version_pin_check()
Knowing which packages were loaded (above) is not the same as knowing which versions. The module separately checks whether the repository pins the exact R/package environment the analysis depended on, via any of:
renv.lock file — parsed for the declared R version and every locked package’s name, version, and source;sessionInfo() / sessioninfo::session_info() text dump — matched by filename (sessionInfo.txt, session_info.txt, and similar), found embedded in a README, or pasted into an R script (usually as comments) when it lists packages, and read for its declared R version and the versions of the packages it lists; orgroundhog::groundhog.library() or checkpoint::checkpoint() date-pin call in the code — a bare library(groundhog)/library(checkpoint) does not count; the actual pinning call (with its date argument) must be present. The date may also be held in a variable or given as snapshot_date, and a call inside a comment does not count. A README in a revdep folder does not count as a record: it gives the R version used to test other packages against an R package.Without one of these, package versions can silently drift between when the analysis was run and any later reproduction attempt — a script that works today may fail or produce different results a year from now if install.packages() pulls whatever the CRAN snapshot happens to be at that time.
Python code gets Python advice. For Python, the module looks for a list of the packages the code needs: requirements*.txt, environment.yml, Pipfile or Pipfile.lock, poetry.lock, uv.lock, or pyproject.toml. These are returned as python_files and python_pinned in $version_pin, next to pinned, which stays about R. Python code without such a list is told to add a requirements.txt, R code keeps the R advice, and a repository with code in both languages needs both. Code in neither language is judged as before. The traffic light and code_version_pinned follow this. In the same way, the advice that goes with an install.packages() call names requirements.txt for Python next to R’s renv.lock and DESCRIPTION.
.code_version_pin_check(all_files, code_text_list = list(r_script_lines))This is also exposed for reproducibility_check’s Docker backend, which uses the declared R version (when found) to pick a matching base image — see docker_use_declared_version in that chapter.
Common problems the module surfaces:
code_abs_path > 0) — e.g. read.csv("C:/Users/me/data.csv") — which will not run on anyone else’s computer.loaded_files_missing > 0) — the script reads a file that was not shared.parse_error == TRUE) — the file is not valid R code.| file_name | code_abs_path | loaded_files_missing | parse_error |
|---|---|---|---|
| test_script.R | 2 | 3 | FALSE |
| bad.R | 1 | 1 | FALSE |
| bad.Rmd | 2 | 1 | FALSE |
| 01.R | 0 | 0 | FALSE |
| 02.R | 0 | 0 | FALSE |
| 03.R | 0 | 0 | FALSE |
| 04.R | 0 | 0 | FALSE |
| 05.R | 0 | 0 | FALSE |
| 06.R | 0 | 0 | FALSE |
| 07.R | 0 | 0 | FALSE |
| 08.R | 0 | 0 | FALSE |
| 09.R | 0 | 0 | FALSE |
| 10.R | 0 | 0 | FALSE |
| 11.R | 0 | 0 | FALSE |
| 12.R | 0 | 0 | FALSE |
| 13.R | 0 | 0 | FALSE |
| 14.R | 0 | 0 | FALSE |
| 15.R | 0 | 0 | FALSE |
| 16.R | 0 | 0 | FALSE |
| 17.R | 0 | 0 | FALSE |
| 18.R | 0 | 0 | FALSE |
| 19.R | 0 | 0 | FALSE |
| 20.R | 0 | 0 | FALSE |
| 21.R | 0 | 0 | FALSE |
| 22.R | 0 | 0 | FALSE |
| 23.R | 0 | 0 | FALSE |
| 24.R | 0 | 0 | FALSE |
| 25.R | 0 | 0 | FALSE |
| good-example.R | 0 | 0 | FALSE |
| test_script.R | 2 | 3 | FALSE |
| Study 1.r | 2 | 3 | FALSE |
| utils.R | 0 | 0 | FALSE |
code_check downloads the code files it will check (download = TRUE, the default) so their contents are read locally rather than streamed from the repository URL each time. They are reused on later runs only if you also pass cache = TRUE; otherwise they are discarded when the session ends. It shares data_check’s size caps (max_file_size, which skips each file larger than it, and max_download_size, a budget per repository that is filled with the smallest files first) and the persistent-cache switch (cache, default FALSE). It also shares repo_check’s local-file options:
# check code in a local folder only, no online lookups
module_run(paper, "code_check", local_path = "path/to/files", local_only = TRUE)
# stream files from their URLs instead of downloading
module_run(paper, "code_check", download = FALSE)See the Local Files chapter for a full walkthrough of checking local code, including cloud-synced folders. If there is no manuscript at all, report_repository("path/to/folder") runs code_check together with repo_check, data_check, and codebook_check on the folder in one step — see Getting Started.
manifest merges the distinct packages loaded across a paper’s code into a metacheck manifest’s code section, alongside whatever data_check already wrote there:
module_run(paper, "code_check", manifest = "path/to/manifest/dir")| Light | Meaning |
|---|---|
| green | at least one file was analysed, no issues were found (missing files, no comments, absolute paths, setwd() calls, install.packages() calls, spread-out libraries, or a parse error), and the repository pins its R/package environment (for R code) and lists its Python packages (for Python code) |
| yellow | at least one of the above issues, or no pinned environment or Python package list was found |
| na | no code files could be analysed (e.g. only JASP or jamovi files were found) |
| Element | What it contains |
|---|---|
$table |
one row per code file, with language, parse_error, code_abs_path, absolute_paths, code_setwd, setwd_calls, code_install_packages, install_packages_calls, percentage_comment, library_max_between, packages, packages_n, loaded_files_missing, and the missing-file names |
$summary_table |
per-paper aggregates (code_n, code_checked, code_abs_path, code_setwd, code_install_packages, code_missing_files, code_min_comments, code_parse_errors, code_packages_n, code_version_pinned) |
$report |
formatted report sections: comments, missing files, absolute paths, working directory, package installation, libraries, packages/dependencies, reproducible environment, and parsable code |
$summary_text |
plain-text bulleted summary of each check |
$version_pin |
the raw output of .code_version_pin_check() (mechanisms found, declared R version(s), any renv.lock package table, and python_files / python_pinned for Python package lists) — used by reproducibility_check()’s Docker backend to pick a matching base image |
Downloaded code files (and any recovered .spv/.smcl/.out/.html-derived syntax files) are cached and reused via the same cache/max_file_size/max_download_size mechanism data_check uses — see the Caching chapter.
The module reports on potential problems. If you want to extend the package to perform additional checks, or make the checks work on other code languages, reach out to the Metacheck development team.
23.6.2 Comments:
code_remove_comments()andcode_line_stats()code_remove_comments()strips comments per language — line comments and block comments in each of R, SAS, SPSS, and Stata (e.g. SAS/* … */blocks and*…;line comments; Stata//end-of-line and/* */blocks).code_line_stats()then reports the total lines, blank lines, code lines, comment lines, and the percentage of comment lines. A file with no comments (percentage_comment == 0) is flagged, because comments are what let a future reader (including the author) understand and reuse the code.