11 Preregistration Check

11.1 What it checks

The prereg_check module finds links to preregistrations on the OSF and AsPredicted in the manuscript, retrieves the preregistration text, and organises its contents into a single standardised template. This makes it much easier for a reviewer to compare what was preregistered against what the manuscript reports.

Note

This module makes live network calls to the OSF and AsPredicted to retrieve preregistration content. You need an internet connection to run the code below. (When this book is built, these calls run against the live services.)

11.2 AsPredicted and the ResearchBox key

Metacheck reads the OSF’s preregistrations through the OSF’s API. It reads an AsPredicted preregistration through ResearchBox’s API when a ResearchBox API key is set, with one request for each preregistration. Without a key, or when the API fails, it reads the preregistration’s web page on AsPredicted. AsPredicted asks for a captcha after several pages, and after that no more pages are read, so a paper set with many AsPredicted links needs the key. Make a key at https://researchbox.org/api and set it with rbox_token():

rbox_token("your-researchbox-key")

Without a key, the report says that the AsPredicted preregistrations were read from the web pages, and how to set a key. ResearchBox allows 200 AsPredicted preregistrations a day for each key. On the website, you can enter your own key under “Your own API keys”. Getting API Tokens explains the keys of the other services.

AsPredicted’s form has had several versions. Metacheck finds each answer by the wording of its question, so it also reads version 1.05, which asks about secondary analyses (in the column exploratory_analyses) where version 2.00 asks about outliers.

11.3 Running the module

Without R

This module also runs on Metacheck’s website: upload a paper under Run Metacheck on a Paper, and tick it under Checks (it is not one of the default checks, because it contacts an online service). The Docker installation offers the same website on your own computer. The website runs the module with its default settings; to change them, run it in R as below.

demopaper() links to two preregistrations, so it is a good example.

paper <- demopaper()
mo <- module_run(paper, "prereg_check")
mo$traffic_light

#> [1] "info"

mo$summary_text

#> [1] "We found 2 preregistrations."

The table has one row per preregistration, with the template fields organised into columns. Here are a few of them:

mo$table[, c("template_name", "title", "id", "sample_size")] |>
  knitr::kable()

template_name	title	id	sample_size
OSF Preregistration	Papercheck Test	48ncu	We randomly assigned 50 scientists to a condition where their manuscript was automatically checked for errors, and 50 scientists to a control condition with a checklist.
AsPredicted	To err is human	by8i8v	NA

The full set of fields the module standardises includes the study design, hypotheses, sample-size rationale, stopping rule, variables, statistical tests, inference criteria, and exclusion criteria:

names(mo$table)

#>  [1] "template_name"              "title"                     
#>  [3] "id"                         "link"                      
#>  [5] "date_created"               "date_modified"             
#>  [7] "date_registered"            "embargo_end_date"          
#>  [9] "ia_url"                     "research_questions"        
#> [11] "study_type"                 "blinding"                  
#> [13] "study_design_overview"      "randomization"             
#> [15] "existing_data"              "data_collection_procedures"
#> [17] "sample_size"                "sample_size_rationale"     
#> [19] "stopping_rule"              "manipulated_variables"     
#> [21] "measured_variables"         "statistical_tests"         
#> [23] "inference_criteria"         "data_exclusion_criteria"   
#> [25] "missing_data_handling"      "exploratory_analyses"      
#> [27] "existing_data_explanation"  "design_dependent_variables"
#> [29] "outliers_and_exclusions"    "additional_comments"       
#> [31] "paper_id"

11.4 Reading a single preregistration’s contents

Because each preregistration becomes a row, you can inspect one in detail by transposing it:

prereg <- mo$table[1, c("template_name", "sample_size", "statistical_tests", "data_exclusion_criteria")]
t(prereg) |>
  knitr::kable()

template_name	OSF Preregistration
sample_size	We randomly assigned 50 scientists to a condition where their manuscript was automatically checked for errors, and 50 scientists to a control condition with a checklist.
statistical_tests	We will perform an independent t-test on the number of mistakes in each group.
data_exclusion_criteria	No exclusions are expected.

11.5 Options and limitations

prereg_check takes the paper and one option, cache. With cache = TRUE, an AsPredicted preregistration that was read before is taken from the saved copy instead of being read again; it is FALSE by default, like Metacheck’s other caches (see Caching).

A link that could not be read, such as a private OSF registration or an AsPredicted page that did not answer, is named in the report with the reason. It does not count as a preregistration that was found. A link to AsPredicted’s home page is not looked up.

The module cannot extract information from:

unstructured preregistration templates (where everything is in one free-text field), or
preregistrations stored as uploaded text documents on the OSF rather than in the structured form.

For the details of which OSF registration templates exist and how they are identified, see the OSF Registration Schemas chapter.

11.6 Notes

If you want Metacheck to support additional preregistration platforms, reach out to the development team.