26  Psych-DS Check

Warning

This module is currently under development, and not available in the main version of Metacheck. To try it out, install the development version of Metacheck:

remotes::install_github("scienceverse/metacheck@data_check")

26.1 What it checks

Psych-DS is a community standard for organising a psychology dataset so that both humans and machines can find their way around it: data live in a data/ folder, analysis code in analysis/, materials in materials/, and a machine-readable dataset_description.json at the root describes the dataset and its variables.

The psychds_check module compares a repository against this standard and reports the compliance gap: for every file, where it currently sits, where the Psych-DS layout says it should go, and what (if anything) is missing at the top level.

It answers three questions:

  • Are the required pieces present (a dataset_description.json, a data/ directory)?
  • Are the recommended pieces present (a README, a CHANGES file)?
  • Which files are misplaced relative to the standard, and where should they move?
Note

This chapter runs offline against a small repository that does not follow the standard, so we can see the gap the module reports.

26.2 A worked example

repo <- file.path(tempdir(), "psychds_check_demo")
dir.create(repo, recursive = TRUE, showWarnings = FALSE)

# a flat repo: data and code sitting at the top level, no metadata
utils::write.csv(data.frame(id = 1:5, score = rnorm(5)),
                 file.path(repo, "study_data.csv"), row.names = FALSE)
writeLines("# analysis\n", file.path(repo, "analysis.R"))
writeLines("A short readme.", file.path(repo, "README.md"))
mo <- module_run(test_paper(), "psychds_check",
                 local_path = repo, local_only = TRUE)

mo$traffic_light
#> [1] "yellow"
cat(mo$summary_text)
#> 
#> -  2 of 3 required Psych-DS items present; 1 missing.
#> -  2 files would need to move to a Psych-DS location.
#> -  2 recommended items missing.

26.3 The compliance table

The table lists each file with its current path and the Psych-DS target path, plus a status saying what needs to happen:

mo$table[, c("file_name", "data_type", "current_path", "target_path", "status")] |>
  knitr::kable()
file_name data_type current_path target_path status
analysis.R code analysis.R analysis/analysis.R move
README.md readme README.md README.md present
study_data.csv data study_data.csv data/source-studydata_data.csv move

A status of move means the file is in the wrong place for a Psych-DS dataset — for example study_data.csv should move into data/, and analysis.R into analysis/. This gives the author a concrete to-do list rather than an abstract “does not comply”.

26.4 The target tree

The most useful part of the report is the target tree: a drawing of the Psych-DS layout the repository should have, with the current files placed where they belong and the missing pieces marked. It turns the compliance table into a single picture you can act on.

The tree uses colour and annotations to show three states:

  • files in red with a ← missing marker are required by Psych-DS but absent (for example dataset_description.json and CHANGES);
  • files annotated (move from …) exist in the repository but need relocating into the standard folder — the annotation tells you exactly where they are now;
  • plain files are already in the right place.

The tree is emitted as HTML in the module’s report. Here it is for our example repository:

tree_html <- grep("<pre", mo$report, value = TRUE)
cat(tree_html)
├── analysis/
│   └── analysis.R (move from analysis.R)
├── data/
│   └── source-studydata_data.csv (move from study_data.csv)
├── CHANGES  ← missing
├── dataset_description.json  ← missing
└── README.md

Reading this top to bottom is a complete migration plan: create the two missing metadata files, move the data file into data/ (renamed to the Psych-DS source-* convention) and the script into analysis/, and the repository will comply. The convert_psychds() function performs exactly these steps automatically.

26.5 The summary counts

mo$summary_table |>
  knitr::kable()
paper_id required_met required_missing recommended_met recommended_missing misplaced_n
bc17b4da2bffba 2 1 1 2 2
Column Meaning
required_met / required_missing required Psych-DS elements present / absent
recommended_met / recommended_missing recommended elements present / absent
misplaced_n files that are not in their Psych-DS location

26.6 The traffic light

Light Meaning
green the repository already follows the Psych-DS layout
yellow some elements are missing or files are misplaced
red required elements (like data/ or the description file) are absent
na there were no files to assess

26.7 From gap report to a compliant dataset: convert_psychds()

psychds_check tells you what is missing; the companion function convert_psychds() acts on it. Given the same repository, it writes out a Psych-DS-compliant copy — moving files into data/, analysis/, and so on, and generating the dataset_description.json metadata from the columns data_check extracted:

res <- report(paper, c("data_check", "codebook_check", "psychds_check"))
convert_psychds(res, overwrite = TRUE)

convert_psychds() can take either a paper object (it runs the checks as needed) or a captured result of report(paper, ...) — when the capture already contains data_check / codebook_check / psychds_check, those outputs are reused rather than re-run, and the OSF listing is not re-queried (refresh_osf = FALSE by default). This is why the recommended pattern is to run the checks once and hand the result to the converter. The output goes to output_dir (default psychds/<paper_id>); an existing directory is skipped unless overwrite = TRUE.

26.7.1 What the converter does, step by step

  1. Resolves the plan. It reuses data_check’s structure (per-file classification) and table (column facets), codebook_check’s labels and identified scales, and psychds_check’s placement plan (current_path → target_path).
  2. Copies every file to its Psych-DS location. Files are resolved to their source by row index, not by name — a paper’s several OSF components can each ship their own demographics.csv, and a name-keyed lookup would silently give every study the first component’s bytes. A non-CSV data source (.xlsx/.sav/.dta/.jasp/…) is written out as a real CSV from the fully-read data frame (a byte copy would be an invalid CSV) and the untouched original is kept beside it, so the release carries both the reusable data and the authored artifact. Data CSVs are written BOM-free, because a UTF-8 BOM makes the first header read as id and mismatches variableMeasured.
  3. Splits multi-study repositories. When data_check assigned study groups (see Data Check), each becomes a self-contained study-<group>/ dataset with its own dataset_description.json. Files belonging to no single study (a whole-repo README/codebook, shared materials) sit at the collection root beside the study folders — following BIDS, which places shared content at the root.
  4. Writes scale definitions as OpenScales OSD files (scales/<code>.osd), and normalises trial-level data into Behaverse paradata/<instrument>.json, cross-linking the two.
  5. Writes the paper’s full text into documentation/, and a provenance logs/ folder (the file manifest, and the check results / module tables when those modules ran), each explained in the root README.

The generated dataset is validated with psychds_validate(), Metacheck’s built-in checker for the standard, so a single-study converted output is guaranteed to pass. convert_psychds() returns (invisibly) a list with output_dir, n_files_copied, n_studies, descriptions, collection, n_scales, n_paradata, fulltext, logs, skipped, and copy_failed.

26.8 The schemas Metacheck writes

convert_psychds() is where several standards come together. Understanding them explains what is in the generated files.

26.8.1 Psych-DS (dataset_description.json)

The heart of a Psych-DS dataset is a root dataset_description.json: schema.org JSON-LD with @type Dataset, schemaVersion "Psych-DS 1.5.1", the paper’s name, author, identifier (DOI), and keywords, and — the substantive part — a variableMeasured array describing every column of every data CSV. Psych-DS requires each CSV column to appear here.

Each entry is a schema.org PropertyValue built by the converter from data_check’s facets and codebook_check’s labels. Because schema.org has no native properties for most psychometric metadata, Metacheck records those under a namespaced metacheck: extension whose fields map onto DDI concepts:

Field Source DDI / schema.org concept
name column name variable name
description codebook label variable label
measurementTechnique identified scale name the instrument a variable was measured with
metacheck:scale scale {name, code, source, confidence} DDI variable group; code cross-references scales/<code>.osd
metacheck:measurementLevel measurement-level facet DDI @classificationLevel
unitText unit facet schema.org unitText (DDI UnitType)
metacheck:concept concept facet DDI Variable → Concept
metacheck:role role facet DDI VariableRole
metacheck:codeList decoded value labels DDI CodeList (one PropertyValue per code → label)
metacheck:missingValues declared missing scheme DDI MissingValues
metacheck:question / metacheck:universe codebook question/filter DDI QuestionText / Universe
minValue/maxValue/metacheck:statistics data_col_stats() schema.org numeric bounds + a stats block
valuePattern sample values observed category pattern

An empty (all-NA) column gets a minimal stub entry naming it and flagging it empty, so the CSV column is documented without inventing statistics.

26.8.2 The collection: collection.json

A multi-study output root is a collection of datasets, not itself a Psych-DS dataset, so it carries no root dataset_description.json. Instead it gets a collection.json — schema.org JSON-LD with @type Collection — whose hasPart indexes every part: each study-<group>/ dataset (with its variable count), the root-level shared files, the paper full text, and the logs. It is deliberately not named dataset_description.json, so the Psych-DS validator (which only ever opens a file of that exact name) never validates the root — you validate each study-*/ folder instead. This is why a multi-study convert is not run through psychds_validate() at the root: the root is correctly not a dataset.

26.8.3 OpenScales OSD (scales/<code>.osd)

Every scale that codebook_check identifies is written as one OpenScales OSD file, flat at scales/<code>.osd. The code is a readable slug derived from the scale name (PANAS → positive_and_negative_affect_schedule) or, for an unnamed block, its column prefix. Each OSD records the scale’s provenance at one of four trust levels (.osd_code_and_provenance()): dictionary (matched a known instrument in Metacheck’s OpenScales-derived scale dictionary), manuscript (a named instrument from the paper), self_generated (an LLM-inferred construct label — not a recognised instrument), or unnamed_block (a coherent same-prefix rating block that could not be named). The variableMeasured entry for each of the scale’s columns cross-references its OSD code, so a reader can jump from a variable to its instrument definition. When a scale matches a dictionary instrument, its reference items (including which are reverse-keyed) are available via the scale_meta / scale_items datasets.

26.8.4 Behaverse trial data (paradata/<instrument>.json)

Trial-level task data (E-Prime, Inquisit, jsPsych, native Behaverse) is not a rectangular dataset — it is one file per participant per block. convert_psychds() normalises all of it into the Behaverse Data Model trial schema (pinned v26.0608, shipped in inst/schema/), writing one paradata/<instrument>.json per instrument with the full response data merged across participants — nothing is deleted. Each file is a TrialData document: an Instrument descriptor plus a Response array, where every response carries the 13 required Behaverse fields (response_id, study_name, agent_id, instrument_id, trial_index, stimulus_id, …) and any optional paradata the source recorded (response_time, correct, stimulus_onset, …). Each source format has its own reader that maps its columns onto this vocabulary — Inquisit’s latency → response_time, jsPsych’s rt → response_time, E-Prime’s <obj>.RT, and so on (convert_behaverse()). The OSD scale file and the paradata file cross-reference each other on the canonical instrument id, so a questionnaire’s scale definition and its raw timing data point at each other.

26.8.5 DDI-Codebook 2.5 (the manifest)

The optional file manifest (data_check’s manifest = argument) is self-describing: its provenance block carries a ddi_mapping recording how each manifest field maps onto a DDI-Codebook 2.5 element (files[].file_name → fileDscr/fileTxt/fileName, files[].status → fileDscr/fileTxt/ProcStat, and so on), alongside the metacheck/R version, platform, and production date needed to reproduce the archive.

26.9 Building a whole data archive

The same machinery scales up from one repository to a data release — a browsable, archivable copy of the data behind an entire corpus of papers, each converted to Psych-DS. This is a good way to turn scattered OSF/GitHub/Zenodo repositories into a single, structured, machine-readable collection.

The recipe is: run the data pipeline over each paper with data_check set to fetch all files, then convert each to Psych-DS. Two data_check options (covered in the Data Check chapter) make this practical for real-world repositories:

  • download = "all" fetches every file, not just the readable data.
  • skip_types = "asset" leaves out stimuli/media (which a release links to rather than hosts), and peek_zips = TRUE looks inside zips and only downloads those containing data.
for (paper in papers) {
  chain <- report(paper,
                  c("data_check", "codebook_check", "psychds_check"),
                  args = list(data_check = list(
                    download = "all", skip_types = "asset", peek_zips = TRUE)))
  convert_psychds(chain,  output_dir = file.path("archive", paper_id(paper), "psychds"))
  convert_codebook(chain, output_dir = file.path("archive", paper_id(paper), "codebook"))
}

26.9.1 Zips are unpacked into the archive

When peek_zips = TRUE, a downloaded zip that contains data is unpacked and its inner data files are placed into the archive individually — so a data.zip becomes real, browsable data/ files rather than an opaque bundle. Two helper functions expose this directly:

  • zip_peek(url) reads a remote zip’s file listing (names and sizes) via an HTTP range request, without downloading it.
  • zip_decision(url) peeks and classifies the contents, returning whether the zip is worth downloading (contains data) or should just be linked (only stimuli).

26.9.2 An archive-level catalogue

A release of many datasets needs a top-level description tying them together. data_catalog() scans a built archive directory and writes a catalog.json — a schema.org DataCatalog listing every dataset with its title, authors, DOI, and file count — plus a catalog.csv for quick inspection. This is the object that makes the release a single discoverable archive (and is harvested by Google Dataset Search), rather than a set of unconnected folders.

data_catalog("archive", papers = papers, name = "My data release")
Tip

For a large corpus, first run data_check with download = "none" and manifest = "..." to write a per-paper file manifest (file names, sizes, types) without downloading anything. Inspecting those manifests lets you choose the size caps knowing exactly which files each value would include or exclude, before committing to a large download.

26.10 What you get back

Element What it contains
$traffic_light green / yellow / red / na
$summary_text plain-text compliance summary
$summary_table the required/recommended/misplaced counts above
$table one row per file: file_name, data_type, current_path, target_path, status
$report formatted report, including the target directory tree with missing pieces marked