10 Preregistration Comparison with RegCheck

10.1 What it checks

The reg_check module compares a paper against its preregistration using RegCheck, an LLM-based tool that judges, dimension by dimension, whether the paper is consistent with what was preregistered. Where the prereg_check module simply retrieves the preregistration, reg_check goes a step further and compares the two documents.

For each dimension — sample size, data source, inclusion and exclusion criteria, hypotheses, manipulated and measured variables, transformations, statistical models, and handling of missing data — RegCheck judges whether the paper:

deviates from the preregistration ("yes"),
is consistent with it ("no"), or
there is insufficient information to tell ("missing").

The comparison is done by a large language model, so the judgements are pointers for manual checking, never automated conclusions. Each comparison makes one LLM call per dimension and can take several minutes per preregistration.

Note

This module sends the full paper and preregistration text to a RegCheck server. The example in this chapter uses the hosted RegCheck app (client = "groq"), which sends text to that server and the Groq LLM provider. You can instead run everything locally so no text leaves your machine — see running RegCheck locally below. All RegCheck servers require an API token in the REGCHECK_API_TOKEN environment variable.

Currently, it is not possible to request API keys through the RegCheck website yet. You can contact Jamie Cummins if you would like to receive an API key.

10.2 Running the module

reg_check builds on prereg_check: it needs the preregistration(s) for a paper. If you run it on its own it calls prereg_check for you first; you can also chain them so the preregistrations are retrieved only once:

paper |>
  module_run("prereg_check") |>
  module_run("reg_check", client = "groq")

Here we run it on demopaper(), which links to two preregistrations (an OSF registration and an AsPredicted), using the hosted app via Groq. (The comparison was run once and its result is loaded here, since each run takes several minutes and sends text to an external service.)

mo <- module_run(demopaper(), "reg_check", client = "groq")

#> [1] "info"

#> [1] "RegCheck compared the paper with 2 preregistrations on 20 dimensions, and flagged 10 potential deviations (6 dimensions not specified in the preregistration)."

The table has one row per dimension per preregistration, with the judgement and supporting detail:

dimension	deviation_judgement	prereg_id
Sample size	yes	48ncu
Data source	yes	48ncu
Inclusion criteria	yes	48ncu
Exclusion criteria	yes	48ncu
Incomplete and missing data	yes	48ncu
Hypotheses	yes	48ncu
Manipulated variables	no	48ncu
Measured variables	no	48ncu
Transformations	missing	48ncu
Statistical models	yes	48ncu

The overall split of judgements:

#> 
#> missing      no     yes 
#>       6       4      10

10.3 Reading a flagged deviation

For any dimension judged a deviation ("yes"), the deviation_information column explains what differs, with quotes from both documents. For example, on sample size RegCheck noticed that although the planned n matched, the paper revealed the data were simulated:

#> While the number of participants per group (50) is consistent between the registration [PREREG_0001, PREREG_0002] and the paper [PAPER_0002], the paper reveals that the data was simulated [PAPER_0003], which deviates from the registration's implication of collecting real data from 100 researchers [PREREG_0002].

These explanations are the useful output: they point a reviewer straight to the sentences to read, rather than asserting that a deviation is necessarily a problem (a preregistered study can deviate for good, disclosed reasons).

10.4 Options

Argument	Default	Effect
`paper`	—	the paper or paperlist to check
`client`	`"ollama"`	which RegCheck backend: `"ollama"` (local), or `"groq"`, `"openai"`, `"deepseek"` (hosted app)
`base_url`	server default	override the RegCheck server URL
`dimensions`	RegCheck default	a data frame of custom dimensions to compare on

The lower-level regcheck_compare() function exposes the same comparison directly on raw text (a paper string plus either a preregistration string or a clinical-trial registration_id); see ?regcheck_compare.

10.5 Running locally

By default (client = "ollama") reg_check uses a RegCheck server running on your own machine with Ollama, so no paper or preregistration text is sent to an external service. This requires a one-time setup of the local RegCheck server (via Docker or Python) and a local Ollama model.

The setup and the helper functions for this — regcheck_setup_local(), regcheck_start_local(), and regcheck_stop_local() — are described in the Local AI with Ollama chapter. Once a local server is running, the only change to the code above is dropping the client argument (or setting client = "ollama"):

module_run(demopaper(), "reg_check")   # client = "ollama" by default