This module has not been validated: its checks have unknown error rates, so evaluate its output carefully. Its identification of psychometric scales is experimental and switched off by default (see Identifying psychometric scales).
26.1 The basics
A codebook is a document that explains what each column in a dataset means — for example, that a column called cond holds a “condition: 0 = control, 1 = treatment”. The codebook_check module looks at the data columns a paper shares (using data_check’s results) and checks, column by column, whether each one is actually described somewhere in a codebook. A column with a matching description is labelled; a column with no description anywhere is unlabelled.
Here is the module running on a small folder with one data file and a codebook that documents two of its three columns, built just for this example:
#>
#> - We parsed 2 variable definitions from codebook/README files.
#> - 2 of 3 data columns (67%) are documented in a codebook; 1 is not.
Running a module always follows the same pattern: module_run(paper, "module_name", ...). Assigning the result to a variable, here mo, lets you look at exactly which columns were and were not documented. The central piece of output is mo$table, with one row per data column and its documentation status:
id comes back unlabelled because the codebook never mentions it, while rt and cond are labelled with the description the codebook gave them — one undocumented column out of three is already enough to turn the light red, so the traffic light is sensitive even to a single gap. A labelled status only means a description exists for that column — it does not mean the description is accurate, complete, or correctly filled in, since codebook_check checks whether a label exists, not whether it is right. It also cannot find documentation that exists but is phrased or formatted in a way the module does not recognise (without an LLM enabled, it only reads structured codebook tables, not free-form prose in a README or methods section). This module does more than this simple matching — it also handles conflicting or ambiguous definitions across multiple studies, richer metadata like value labels and missing-value codes, and identifying named psychometric scales — covered in the rest of this chapter.
26.2 What it checks
Shared data is only reusable if others can tell what each column means. The codebook_check module measures documentation coverage: for every data column extracted by data_check, it asks whether that column is described in a codebook (a data dictionary) shared alongside the data.
It reports, for each column, one of these statuses:
labelled — the column is matched to an entry in a codebook file.
unlabelled — no documentation was found for the column.
conflicting_definition — the codebook contains more than one, disagreeing, description for the column within its own study group.
ambiguous_experiment — the column only matches codebook entries scoped to a different study group than its own, so no applicable description could be chosen.
llm — the label was inferred by an LLM (only when one is enabled).
There is no single "conflicted" value — conflicting_definition and ambiguous_experiment are the two distinct conflict sub-types, and code that filters on label_status should check for both.
It also reports unused codebook entries: variables documented in the codebook that do not correspond to any column in the data.
Note
This chapter runs offline against a small folder containing a data file and a codebook. If there is no manuscript at all, report_repository("path/to/folder") runs codebook_check together with repo_check, code_check, and data_check on the folder in one step — see Getting Started.
NoteWithout R
On Metacheck’s website, codebook_check is not part of Run Metacheck on a Repository, which runs repo_check, code_check, and data_check. To run it there, upload a paper under Run Metacheck on a Paper and tick it under Checks: it then checks the repositories the paper links to. The Docker installation offers the same website on your own computer. For a folder without a paper, run it in R with report_repository(), as described below.
26.3 A worked example
We create a study with a codebook.csv documenting two of its three columns:
repo<-file.path(tempdir(), "codebook_check_demo")dir.create(file.path(repo, "data"), recursive =TRUE, showWarnings =FALSE)utils::write.csv(data.frame(id =1:6, rt =c(510, 490, 505, 512, 498, 530), cond =c(0, 1, 0, 1, 0, 1)),file.path(repo, "data", "study.csv"), row.names =FALSE)# a codebook documenting rt and cond (but not id)writeLines(c("variable,description","rt,reaction time in milliseconds","cond,condition: 0 = control, 1 = treatment"),file.path(repo, "codebook.csv"))
Here rt and cond are labelled (matched to the codebook), while id is unlabelled — there was no codebook entry for it.
Matching is not purely literal: names are normalised (case, separators, and simple stemming) so that reaction_time in the data matches Reaction Time in the codebook. This tolerance is important because codebooks and data files rarely use byte-identical names.
26.5 Richer variable metadata
A good codebook records more than a one-line label. Following the DDI standard, codebook_check also captures three further per-variable properties whenever a source supplies them, and carries them onto the matched data columns:
Column
DDI concept
What it holds
value_labels
CodeList / ValueDomain
the code → label mapping, e.g. 1 = "Male", 2 = "Female"
missing_values
MissingValues
which codes denote missingness (e.g. -99 = "Refused")
question
QuestionText
the question the respondent was asked
These come from two sources for free. From SPSS/Stata/SAS files, codebook_check reads the embedded value labels and declared missing values that haven attaches to each column — information most tools discard. From a text codebook, it parses a “values” / “coding” column (understanding common encodings like 1 = Male; 2 = Female or 0: no | 1: yes) and a dedicated question column.
Let’s document sex with a coding column and a question:
repo2<-file.path(tempdir(), "codebook_values_demo")dir.create(file.path(repo2, "data"), recursive =TRUE, showWarnings =FALSE)utils::write.csv(data.frame(sex =c(1, 2, 1, 2, -99), age =c(20, 30, 40, 50, 25)),file.path(repo2, "data", "study.csv"), row.names =FALSE)writeLines(c("variable,description,values,question","sex,Participant sex,1 = Male; 2 = Female; -99 = Refused,What is your sex?","age,Age in years,,How old are you?"),file.path(repo2, "data", "codebook.csv"))mo2<-module_run(test_paper(), "codebook_check", local_path =repo2, local_only =TRUE)mo2$table[, c("column_name", "value_labels", "missing_values", "question")]|>knitr::kable()
column_name
value_labels
missing_values
question
sex
{“1”:“Male”,“2”:“Female”,“-99”:“Refused”}
{“-99”:“Refused”}
What is your sex?
age
NA
NA
How old are you?
The code list is stored as a compact JSON string ({"1":"Male","2":"Female","-99":"Refused"}); a code whose label reads as missingness (Refused, N/A, Prefer not to answer, …) is additionally recorded in missing_values, so a sentinel like -99 is flagged as a missing code rather than a real value.
All three properties are designed to be exported into the Psych-DS metadata by convert_psychds() (not yet available): the code list becomes a schema.org code list (one PropertyValue per code), and the missing scheme and question become namespaced metacheck: fields, so they travel with the converted dataset.
matched columns with a single vs. a contradictory definition
codebook_var_n
entries found in the codebook(s)
unused_var_n
codebook entries not used by any column
26.7 The traffic light
Light
Meaning
green
every column is documented, no column has conflicting definitions, and no data value falls outside the values the codebook documents
yellow
at least 80% of columns are documented (or the only problem is conflicting definitions), or the data hold values the codebook does not list
red
no codebook was found, or fewer than 80% of columns are documented
na
there were no data columns to check
26.8 Using an LLM
Many repositories document their variables in prose — a README paragraph, a methods section, or a non-tabular codebook — rather than a tidy variable,description table. With llm_use(TRUE), codebook_check can read those unstructured sources and propose labels for otherwise unlabelled columns, marking them with the llm status so you can tell inferred labels from author-supplied ones. For large repositories the LLM calls are batched to stay within request limits. On Metacheck’s website and in the Docker installation, only the power check uses a language model, so this only happens when you run codebook_check in R.
The codebook_max_calls argument (default 40) caps how many LLM calls a single tier will make — the number of text blocks parsed from one unstructured codebook file, and the number of distinct survey layouts sent for scale identification (only with extract_scales = TRUE, see below). If a tier would need more than this, it is skipped rather than truncated, and the report says how many were needed; raise codebook_max_calls to include them.
With extract_scales = TRUE, codebook_check also tries to recognise the psychometric scales (and behavioural tasks) in the data. This part of the module is experimental and has not been validated, so it is switched off by default. It looks for blocks of Likert-type items — runs of adjacent integer columns (3–11 response options) that share a variable-name prefix, five items or more — and works out which published instrument each block is (PANAS, the Rosenberg Self-Esteem Scale, and so on). Without a language model, a block is named when its variable-name prefix matches an instrument in Metacheck’s scale dictionary. With an LLM enabled, the model is also asked to name each block from the paper’s text and the item wording from the codebook, and, as a last resort, to describe what an unnamed block measures.
Each identified scale is recorded in two new columns of the $table:
scale — the instrument name (e.g. "PANAS"), for every item column of the block;
scale_confidence — high, medium, or low (low-confidence and unrecognised blocks are left blank rather than guessed).
The report gains a Scales section listing what was found. When a block looks like a scale but could not be named, the module says so and gives concrete advice — because a scale that a language model cannot identify is usually one a human reader will struggle with too:
name variables consistently after the instrument (panas_1 … panas_10) rather than Q1, V3, or item5;
document the item wording in a codebook or embedded value labels — the item text is what identifies a scale;
state the scale name and reference, the response options, and any reverse-coded items.
With extract_scales = FALSE (the default), the scale and scale_confidence columns are still present but empty, and the report has no Scales section. With extract_scales = TRUE but no LLM, the report notes that naming scales from the manuscript needs an LLM, while the dictionary still names instruments whose abbreviation matches a known scale.
26.9.1 How scales are recorded (and why)
No metadata standard metacheck writes to has a first-class “scale” concept. Psych-DS describes each variable in isolation, and schema.org — its foundation — has no property for grouping variables into a construct. The closest existing concept is DDI Codebook’s variable group with type="analysis" (“variables combined into the same index”), but that is a separate XML format psychologists rarely use.
Rather than invent a proprietary field or emit a second file, metacheck extends the Psych-DS metadata in place:
the scale name goes in schema.org’s own measurementTechnique property — its intended use is exactly to name the instrument a variable was measured with;
the grouping (which variables belong to the scale, and the confidence) goes in a namespaced metacheck:scale / metacheck:scales extension, since schema.org has no native grouping and Psych-DS tolerates namespaced JSON-LD.
convert_psychds() and convert_codebook() will both emit these, so the scale information travels with the converted dataset; neither is available yet. When the identified names are available to data_check, the data-quality checks reuse them, labelling their careless-responding findings with the real scale (e.g. “PANAS”) instead of a bare column prefix; usually they are not, because codebook_check runs after data_check (see Codebook labels).
# extract_scales = TRUE switches scale identification on; the LLM is optional# and adds naming from the paper's textllm_use(TRUE)llm_model("groq/openai/gpt-oss-20b")cc<-module_run(test_paper(), "codebook_check", local_path =repo, local_only =TRUE, extract_scales =TRUE)# the identified scales, one row per (column, scale)unique(cc$table[!is.na(cc$table$scale), c("column_name", "scale", "scale_confidence")])
26.9.2 OpenScales OSD export and provenance
Beyond the two $table columns, codebook_check also assembles a ready-to-write OpenScales OSD object for each named scale, returned in $scales_osd. convert_psychds() is designed to write these out as scales/<code>.osd files in the converted dataset, but is not yet available. The code is a readable slug from the scale name (PANAS → positive_and_negative_affect_schedule) or, for an unnamed block, its column prefix.
Each OSD records where the scale identification came from, at one of four honestly-labelled trust levels — this matters because a scale identified from a curated dictionary is far more reliable than one a language model inferred from item wording:
scale_source
Meaning
matched
matched a known instrument in Metacheck’s scale dictionary (OpenScales-derived or curated)
task_matched
matched a known task (rather than a self-report scale) in the same dictionary
manuscript
a real instrument named in the paper, but not in the registry
self_generated
an LLM-inferred construct label — not a recognised named instrument
unnamed_block
a coherent same-prefix rating block, detected but not named
Data files with embedded scale metadata are handled too: a JASP (.jasp) or jamovi (.omv) file carries haven-style value labels just like a .sav, so codebook_check harvests those labels directly, and their bundled analyses are recovered separately (see the JASP/jamovi handling in Data Check).
26.10 What you get back
Element
What it contains
$traffic_light
green / yellow / red / na
$summary_text
plain-text coverage summary
$summary_table
the counts described above
$table
one row per column: column_name, label, label_status, codebook_variable, value_labels, missing_values, question, scale, scale_confidence, scale_source, …
$codebook_vars
the variable definitions found in the codebook(s)
$scale_violations
one row per column holding values its codebook does not list (the check for values outside a rating scale, see Data Quality Checks)
$scales_osd
the identified scales in OpenScales OSD form (only with extract_scales = TRUE)
$report
formatted report of documented vs. undocumented columns, plus identified scales and improvement advice
The convert_codebook() function will turn this output into a fully labelled data frame and a runnable codebook document. It is not yet available in the released version or in dev.
The psychds_check module checks the repository’s structure against a standard, complementing this check of its documentation.