code_lang(c("analysis.R", "clean.do", "model.sas", "survey.sps", "study.jasp"))22 Code Check
22.1 What it checks
The code_check module analyses the code files shared with a paper. Building on the file list from repo_check, it identifies code files, then checks each for signs of reproducibility problems:
- whether R-type files parse without errors;
- whether files contain comments;
- whether they use absolute file paths (which break on other machines);
- whether they call
setwd()(which hardcodes an assumption about the working directory); - whether the files they try to load are actually present in the repository;
- whether libraries are loaded in multiple places rather than in a single block near the top of the script; and
- whether the repository pins its R/package environment (
renv.lock, asessionInfo()dump, or agroundhog/checkpointdate-pin call), so package versions do not silently drift between the original analysis and any later reproduction attempt.
The module reports on potential problems. An absolute path or a missing file is not always an error — but it is usually worth a second look if you want the analysis to be reproducible. The checks are regular-expression based, so they can miss things or produce false positives on unusual scripts; the package was validated internally on papers published in Psychological Science.
Note
This module makes live network calls to retrieve and inspect code files from the linked repositories. You need an internet connection to run the code below.
22.2 Languages: code_lang()
The module first labels every file with its language using code_lang(), which is name-based (by extension):
| Language | Extensions | Analysed? |
|---|---|---|
| R |
.R, .Rmd, .qmd
|
yes (parsed, commented, paths, setwd(), libraries, loaded files) |
| Python |
.py, .ipynb
|
yes (except parsing) |
| SAS | .sas |
yes (except parsing) |
| SPSS | .sps |
yes (except parsing) |
| Stata |
.do, .ado
|
yes (except parsing) |
| Mplus | .inp |
yes (except parsing) |
| MATLAB | .m |
yes (except parsing) |
| JASP | .jasp |
listed only |
| jamovi | .omv |
listed only |
A .ipynb notebook is JSON, not a plain script: its code cells are extracted from the source array of each cell (the same way .Rmd/.qmd are purled to R code below) before any text-based check runs. .py and .m are plain text and are checked on the same terms as SAS/SPSS/Stata: every text-based check applies, but running the code stays out of scope (that is reproducibility_check’s job, and it is R-only).
Some file types are never code themselves but contain the exact syntax that produced a piece of output, and the module recovers that syntax as a proper sibling code file before checking it: an SPSS .spv (rendered output) has its syntax recovered as a .sps file, a Stata .smcl log has its echoed commands recovered as a .do file, an Mplus .out file has its “INPUT INSTRUCTIONS” section recovered as an .inp file, and an .html file is sniffed to tell a rendered R Markdown/Quarto report (whose R source is recovered) apart from an unrelated web page.
analysis.R clean.do model.sas survey.sps study.jasp
"R" "Stata" "SAS" "SPSS" "JASP"A .jasp or .omv file bundles a dataset with its analyses, but it is a binary (zip) archive, so none of the text-based checks apply. These files are counted and listed for completeness but not opened here (their data are read by data_check, and their analysis syntax recovered by read_jasp() / read_omv()). Only R-type files are parsed — the other text languages are checked for everything except parse errors.
22.3 Reading code: code_read() and code_extract_r()
Code is read with code_read(), which guesses the file encoding (readr::guess_encoding()), reads the lines, and converts everything to UTF-8, replacing invalid bytes so a mixed-encoding script never crashes the checks. For R Markdown and Quarto files, code_extract_r() first purls the document (knitr::purl) down to just its R code, so prose and chunk options are not mistaken for code. The module detects an Rmd/qmd file by its leading --- YAML fence and routes it through code_extract_r() automatically.
file_path <- demofile("qmd")
code_text <- code_extract_r(file_path) # R code only, prose stripped22.4 Running the module
paper <- demopaper()
mo <- module_run(paper, "code_check")
mo$traffic_light#> [1] "yellow"cat(mo$summary_text)#>
#> - We found 1 R, 0 SAS, 0 SPSS, 0 Stata, and 0 JASP code file.
#> - All your code files had comments.
#> - 1 file loaded in the code was missing in the repository.
#> - Absolute file paths were found.
#> - No setwd() calls were found.
#> - All libraries/imports were loaded in one block.
#> - The code loaded 1 distinct package.
#> - No parsing issues of R-type files were found.The table has one row per code file, with the reproducibility checks as columns:
22.5 The checks, function by function
Each check is a small exported helper, so you can run it on your own code independently of the module.
22.5.1 Parsing: code_parse_r()
For R-type files, code_parse_r() attempts to parse() the code and captures any syntax error (rewriting <text> in the message to line for readability). Only R is parsed; other languages report NA (not assessed). A parse error means the file is not valid R and would not run at all.
22.5.3 Absolute paths: code_abs_path()
code_abs_path() searches the (comment-free) code for quoted strings that are absolute filesystem paths: a Windows drive path (C:/…, D:\…), a Unix root path (/Users/…), a home path (~/…), or a UNC network path (\\host\share\…). These break on anyone else’s computer. The UNC branch is deliberately strict — it requires the real \\host\share shape — because a bare backslash string is far more often a regex escape ("\\d+") than a network path. The module notes these are potential false positives in code like paste0(dir, "/file.csv").
code_abs_path(c("file <- 'C:/Users/lakens/data.csv'",
"tmp <- '/home/lakens/out.html'",
"rel <- './clean.csv'"))#> # A tibble: 2 × 2
#> abs_path line
#> <chr> <int>
#> 1 C:/Users/lakens/data.csv 1
#> 2 /home/lakens/out.html 2
22.5.4 Working directory: code_setwd()
code_setwd() finds R setwd() calls (on comment-free code). setwd() hardcodes an assumption about the working directory — often an absolute path on the author’s own machine — so the script breaks when run anywhere else. This check is R-only, since setwd() is an R construct.
code_setwd(c("setwd('C:/Users/lakens/project')", "d <- read.csv('x.csv')"))#> # A tibble: 1 × 2
#> setwd_call line
#> <chr> <int>
#> 1 setwd('C:/Users/lakens/project') 1
22.5.5 Library placement: code_library_lines()
code_library_lines() returns the line numbers on which imports/library loads occur, with language-specific patterns (R library/require/renv::install/p_load; SAS %include/libname/filename; SPSS INSERT/BEGIN PROGRAM; Stata do/net install/ssc install). The module computes the maximum gap between successive import lines: if they are more than 3 non-comment lines apart, the file is flagged, because best practice is to load all dependencies in one block near the top so a script’s requirements are easy to see.
code_library_lines(c("library(dplyr)", "x <- 1", "library(tidyr)"), "R")#> # A tibble: 2 × 2
#> code line
#> <chr> <int>
#> 1 library(dplyr) 1
#> 2 library(tidyr) 3
22.5.6 Missing loaded files: code_file_refs()
code_file_refs() extracts the files a script reads or loads — from read/load calls (read.csv, read_csv, fread, readRDS, load, fromJSON, source, and their SAS/SPSS/Stata equivalents like proc import, infile, GET DATA /FILE=, use, import delimited) — by capturing the quoted filename argument. The module compares each referenced file (by basename) against the files actually present in the same repository; anything not found is reported as a missing loaded file, so a reader can check whether the input data were shared. There can be valid reasons a file is not shared (e.g. it holds personal data), which the module cannot evaluate, so it always reports rather than judges.
code_file_refs(c("source('functions.R')", "d <- read.csv('study.csv')"), "R")#> [1] "functions.R" "study.csv"
22.5.7 Packages loaded: code_library_names()
code_library_names() extracts the names of the packages/libraries a file loads (R library()/require()/::; Python import/from … import). The module unions and de-duplicates these across all of a paper’s code files (code_packages()), giving the full list of dependencies the analysis relies on — with no version information, since that is static analysis of source text, not an installed environment.
code_library_names(c("library(dplyr)", "import numpy as np"), "R")#> package source line
#> 1 dplyr library 1
22.5.8 Reproducible environment: .code_version_pin_check()
Knowing which packages were loaded (above) is not the same as knowing which versions. The module separately checks whether the repository pins the exact R/package environment the analysis depended on, via any of:
- an
renv.lockfile — parsed for the declared R version and every locked package’s name, version, and source; - a
sessionInfo()/sessioninfo::session_info()text dump — matched by filename (sessionInfo.txt,session_info.txt, and similar) or found embedded in a README, and read for its declared R version; or - a
groundhog::groundhog.library()orcheckpoint::checkpoint()date-pin call in the code — a barelibrary(groundhog)/library(checkpoint)does not count; the actual pinning call (with its date argument) must be present.
Without one of these, package versions can silently drift between when the analysis was run and any later reproduction attempt — a script that works today may fail or produce different results a year from now if install.packages() pulls whatever the CRAN snapshot happens to be at that time.
.code_version_pin_check(all_files, code_text_list = list(r_script_lines))This is also exposed to reproducibility_check’s Docker backend, which uses the declared R version (when found) to pick a matching base image — see docker_use_declared_version in that chapter.
22.6 A clean example and one with problems
Common problems the module surfaces:
-
Absolute paths (
code_abs_path > 0) — e.g.read.csv("C:/Users/me/data.csv")— which will not run on anyone else’s computer. -
Missing loaded files (
loaded_files_missing > 0) — the script reads a file that was not shared. -
Parse errors (
parse_error == TRUE) — the file is not valid R code.
22.7 Downloading and options
code_check downloads the code files it will check into the shared cache (download = TRUE, the default) so their contents are read locally and reused on later runs, rather than streamed from the repository URL each time. It shares data_check’s size caps (max_file_size, max_download_size — an upfront, all-or-nothing gate per repository) and the persistent-cache switch (cache, default FALSE). It also shares repo_check’s local-file options:
# check code in a local folder only, no online lookups
module_run(paper, "code_check", local_path = "path/to/files", local_only = TRUE)
# stream files from their URLs instead of downloading
module_run(paper, "code_check", download = FALSE)See the Local Files chapter for a full walkthrough of checking local code, including cloud-synced folders.
manifest merges the distinct packages loaded across a paper’s code into a metacheck manifest’s code section, alongside whatever data_check already wrote there:
module_run(paper, "code_check", manifest = "path/to/manifest/dir")22.8 The traffic light and what you get back
| Light | Meaning |
|---|---|
| green | at least one file was analysed, no issues were found (missing files, no comments, absolute paths, setwd() calls, spread-out libraries, or a parse error), and the repository pins its R/package environment |
| yellow | at least one of the above issues, or no pinned environment was found |
| na | no code files could be analysed (e.g. only JASP/jamovi files were found) |
| Element | What it contains |
|---|---|
$table |
one row per code file, with language, parse_error, code_abs_path, absolute_paths, code_setwd, setwd_calls, percentage_comment, library_max_between, packages, packages_n, loaded_files_missing, and the missing-file names |
$summary_table |
per-paper aggregates (code_n, code_checked, code_abs_path, code_setwd, code_missing_files, code_min_comments, code_parse_errors, code_packages_n, code_version_pinned) |
$report |
formatted report sections: comments, missing files, absolute paths, working directory, libraries, packages/dependencies, reproducible environment, and parsable code |
$summary_text |
plain-text bulleted summary of each check |
$version_pin |
the raw output of .code_version_pin_check() (mechanisms found, declared R version(s), and any renv.lock package table) — also used internally by reproducibility_check()’s Docker backend to pick a matching base image |
Downloaded code files (and any recovered .spv/.smcl/.out/.html-derived syntax files) are cached and reused via the same cache/max_file_size/max_download_size mechanism data_check uses — see the Caching chapter.
22.9 Notes
The module reports on potential problems. If you want to extend the package to perform additional checks, or make the checks work on other code languages, reach out to the Metacheck development team.
22.5.2 Comments:
code_remove_comments()andcode_line_stats()code_remove_comments()strips comments per language — line comments and block comments in each of R, SAS, SPSS, and Stata (e.g. SAS/* … */blocks and*…;line comments; Stata//end-of-line and/* */blocks).code_line_stats()then reports the total lines, blank lines, code lines, comment lines, and the percentage of comment lines. A file with no comments (percentage_comment == 0) is flagged, because comments are what let a future reader (including the author) understand and reuse the code.