25  Codebook Check

Warning

This module is currently under development, and not available in the main version of Metacheck. To try it out, install the development version of Metacheck:

remotes::install_github("scienceverse/metacheck@data_check")

25.1 What it checks

Shared data is only reusable if others can tell what each column means. The codebook_check module measures documentation coverage: for every data column extracted by data_check, it asks whether that column is described in a codebook (a data dictionary) shared alongside the data.

It reports, for each column, one of these statuses:

• labelled — the column is matched to an entry in a codebook file.
• unlabelled — no documentation was found for the column.
• conflicted — the codebook contains more than one, disagreeing, description.
• llm — the label was inferred by an LLM (only when one is enabled).

It also reports unused codebook entries: variables documented in the codebook that do not correspond to any column in the data.

Note

This chapter runs offline against a small folder containing a data file and a codebook.

25.2 A worked example

We create a study with a codebook.csv documenting two of its three columns:

repo <- file.path(tempdir(), "codebook_check_demo")
dir.create(file.path(repo, "data"), recursive = TRUE, showWarnings = FALSE)

utils::write.csv(
  data.frame(id = 1:6,
             rt = c(510, 490, 505, 512, 498, 530),
             cond = c(0, 1, 0, 1, 0, 1)),
  file.path(repo, "data", "study.csv"), row.names = FALSE)

# a codebook documenting rt and cond (but not id)
writeLines(c("variable,description",
             "rt,reaction time in milliseconds",
             "cond,condition: 0 = control, 1 = treatment"),
           file.path(repo, "codebook.csv"))

mo <- module_run(test_paper(), "codebook_check",
                 local_path = repo, local_only = TRUE)

mo$traffic_light

#> [1] "red"

cat(mo$summary_text)

#> 
#> -  We parsed 2 variable definitions from codebook/README files.
#> -  2 of 3 data columns (67%) are documented in a codebook; 1 is not.

25.3 The coverage table

The table gives one row per data column with its documentation status and, where matched, the label found:

mo$table[, c("column_name", "label_status", "label", "codebook_variable")] |>
  knitr::kable()

column_name	label_status	label	codebook_variable
id	unlabelled	NA	NA
rt	labelled	reaction time in milliseconds	rt
cond	labelled	condition: 0 = control	cond

Here rt and cond are labelled (matched to the codebook), while id is unlabelled — there was no codebook entry for it.

Matching is not purely literal: names are normalised (case, separators, and simple stemming) so that reaction_time in the data matches Reaction Time in the codebook. This tolerance is important because codebooks and data files rarely use byte-identical names.

25.4 Richer variable metadata

A good codebook records more than a one-line label. Following the DDI standard, codebook_check also captures four further per-variable properties whenever a source supplies them, and carries them onto the matched data columns:

Column	DDI concept	What it holds
value_labels	CodeList / ValueDomain	the code → label mapping, e.g. 1 = "Male", 2 = "Female"
missing_values	MissingValues	which codes denote missingness (e.g. -99 = "Refused")
question	QuestionText	the question the respondent was asked
universe	Universe	the population / filter the variable applies to

These come from two sources for free. From SPSS/Stata/SAS files, codebook_check reads the embedded value labels and declared missing values that haven attaches to each column — information most tools discard. From a text codebook, it parses a “values” / “coding” column (understanding common encodings like 1 = Male; 2 = Female or 0: no | 1: yes) and dedicated question and universe columns.

Let’s document sex with a coding column and a question:

repo2 <- file.path(tempdir(), "codebook_values_demo")
dir.create(file.path(repo2, "data"), recursive = TRUE, showWarnings = FALSE)

utils::write.csv(
  data.frame(sex = c(1, 2, 1, 2, -99), age = c(20, 30, 40, 50, 25)),
  file.path(repo2, "data", "study.csv"), row.names = FALSE)

writeLines(c("variable,description,values,question",
             "sex,Participant sex,1 = Male; 2 = Female; -99 = Refused,What is your sex?",
             "age,Age in years,,How old are you?"),
           file.path(repo2, "data", "codebook.csv"))

mo2 <- module_run(test_paper(), "codebook_check",
                  local_path = repo2, local_only = TRUE)
mo2$table[, c("column_name", "value_labels", "missing_values", "question")] |>
  knitr::kable()

column_name	value_labels	missing_values	question
sex	{“1”:“Male”,“2”:“Female”,“-99”:“Refused”}	{“-99”:“Refused”}	What is your sex?
age	NA	NA	How old are you?

The code list is stored as a compact JSON string ({"1":"Male","2":"Female","-99":"Refused"}); a code whose label reads as missingness (Refused, N/A, Prefer not to answer, …) is additionally recorded in missing_values, so a sentinel like -99 is flagged as a missing code rather than a real value.

All four properties are exported into the Psych-DS metadata by convert_psychds(): the code list becomes a schema.org code list (one PropertyValue per code), and the missing scheme, question, and universe become namespaced metacheck: fields — so they travel with the converted dataset.

25.5 The summary counts

mo$summary_table |>
  knitr::kable()

paper_id	column_n	matched_n	unmatched_n	clean_n	conflicted_n	codebook_var_n	unused_var_n
787ea4c265ae44	3	2	1	2	0	2	0

Column	Meaning
column_n	total data columns
matched_n	columns matched to a codebook entry
unmatched_n	columns with no documentation
clean_n / conflicted_n	matched columns with a single vs. a contradictory definition
codebook_var_n	entries found in the codebook(s)
unused_var_n	codebook entries not used by any column

25.6 The traffic light

Light	Meaning
green	all (or nearly all) columns are documented
yellow	some columns are undocumented
red	no codebook was found, or documentation coverage is very low

25.7 Using an LLM

Many repositories document their variables in prose — a README paragraph, a methods section, or a non-tabular codebook — rather than a tidy variable,description table. With llm_use(TRUE), codebook_check can read those unstructured sources and propose labels for otherwise unlabelled columns, marking them with the llm status so you can tell inferred labels from author-supplied ones. For large repositories the LLM calls are batched to stay within request limits.

llm_use(TRUE)
llm_model("groq/openai/gpt-oss-20b")
module_run(test_paper(), "codebook_check",
           local_path = repo, local_only = TRUE)

The codebook_max_calls argument (default 40) caps how many LLM calls a single tier will make — the number of text blocks parsed from one unstructured codebook file, and the number of distinct survey layouts sent for scale identification. If a tier would need more than this, it is skipped rather than truncated, and the report says how many were needed; raise codebook_max_calls to include them.

See the LLMs chapter for model configuration.

25.8 Identifying psychometric scales

With an LLM enabled, codebook_check also tries to recognise the psychometric scales in the data. It looks for blocks of Likert-type items — runs of adjacent integer columns (3–11 response options) that share a variable-name prefix, five items or more — and asks the model which published instrument each block is (PANAS, the Rosenberg Self-Esteem Scale, and so on), using the item wording from the codebook where available and the variable-name prefix otherwise.

Each identified scale is recorded in two new columns of the $table:

• scale — the instrument name (e.g. "PANAS"), for every item column of the block;
• scale_confidence — high, medium, or low (low-confidence and unrecognised blocks are left blank rather than guessed).

The report gains a Scales section listing what was found. When a block looks like a scale but could not be named, the module says so and gives concrete advice — because a scale that a language model cannot identify is usually one a human reader will struggle with too:

• name variables consistently after the instrument (panas_1 … panas_10) rather than Q1, V3, or item5;
• document the item wording in a codebook or embedded value labels — the item text is what identifies a scale;
• state the scale name and reference, the response options, and any reverse-coded items.

Without an LLM, the columns are still present but empty, and the report notes that identification was skipped.

25.8.1 How scales are recorded (and why)

No metadata standard metacheck writes to has a first-class “scale” concept. Psych-DS describes each variable in isolation, and schema.org — its foundation — has no property for grouping variables into a construct. The closest existing concept is DDI Codebook’s variable group with type="analysis" (“variables combined into the same index”), but that is a separate XML format psychologists rarely use.

Rather than invent a proprietary field or emit a second file, metacheck extends the Psych-DS metadata in place:

• the scale name goes in schema.org’s own measurementTechnique property — its intended use is exactly to name the instrument a variable was measured with;
• the grouping (which variables belong to the scale, and the confidence) goes in a namespaced metacheck:scale / metacheck:scales extension, since schema.org has no native grouping and Psych-DS tolerates namespaced JSON-LD.

convert_psychds() and convert_codebook() both emit these, so the scale information travels with the converted dataset. The Data Validate module reuses the identified names, labelling its careless-responding findings with the real scale (e.g. “PANAS”) instead of a bare column prefix.

llm_use(TRUE)
llm_model("groq/openai/gpt-oss-20b")
cc <- module_run(test_paper(), "codebook_check",
                 local_path = repo, local_only = TRUE)

# the identified scales, one row per (column, scale)
unique(cc$table[!is.na(cc$table$scale), c("column_name", "scale", "scale_confidence")])

25.8.2 OpenScales OSD export and provenance

Beyond the two $table columns, codebook_check also assembles a ready-to-write OpenScales OSD object for each named scale, returned in $scales_osd. convert_psychds() writes these out as scales/<code>.osd files in the converted dataset (see the Psych-DS Check chapter). The code is a readable slug from the scale name (PANAS → positive_and_negative_affect_schedule) or, for an unnamed block, its column prefix.

Each OSD records where the scale identification came from, at one of four honestly-labelled trust levels — this matters because a scale identified from a curated dictionary is far more reliable than one a language model inferred from item wording:

scale_source	Meaning
dictionary	matched a known instrument in Metacheck’s scale dictionary (OpenScales-derived or curated)
manuscript	a real instrument named in the paper, but not in the registry
self_generated	an LLM-inferred construct label — not a recognised named instrument
unnamed_block	a coherent same-prefix rating block, detected but not named

When a scale matches a dictionary instrument, Metacheck can also cross-reference the published original — how many items it should have and which are reverse-keyed — via the packaged scale_meta / scale_items / scale_scoring datasets, so a detected block can be compared against the reference instrument.

Data files with embedded scale metadata are handled too: a JASP (.jasp) or jamovi (.omv) file carries haven-style value labels just like a .sav, so codebook_check harvests those labels directly, and their bundled analyses are recovered separately (see the JASP/jamovi handling in Data Check).

25.9 What you get back

Element	What it contains
$traffic_light	green / yellow / red / na
$summary_text	plain-text coverage summary
$summary_table	the counts described above
$table	one row per column: column_name, label, label_status, codebook_variable, value_labels, missing_values, question, universe, scale, scale_confidence, …
$report	formatted report of documented vs. undocumented columns, plus identified scales and improvement advice

25.10 Related

• Data Check supplies the columns being documented.
• The convert_codebook() function turns this output into a fully labelled data frame and a runnable codebook document — see the Creating a Report workflow.
• Psych-DS Check checks the repository’s structure against a standard, complementing this check of its documentation.