download.file(
"https://raw.githubusercontent.com/Lakens/stroop_effect_paper/main/output/analysis.pdf",
destfile = "good_paper.pdf", mode = "wb"
)3 Your First Metacheck
This chapter walks you through the four things you will do most often with Metacheck, using two short example papers, so that you have a working example before you read a month in depth overview of how the modules work. If you have never used R before, that is fine: every line below is explained, and you can copy and paste each code chunk into the R console one at a time.
We will use two small manuscripts, both about the same simple study (whether people are slower to name the ink color of a word when the word itself names a different color — the “Stroop effect”):
- A good paper: https://github.com/Lakens/stroop_effect_paper, which reports its statistics correctly and shares its data and code openly.
- A bad paper: https://github.com/Lakens/stroop_effect_paper_bad, a version of the same study with a number of common mistakes deliberately introduced, so you can see what Metacheck flags when something is wrong.
3.1 Step 1: Read in a paper
Metacheck works with PDF files. The function convert() turns a PDF into a structured format Metacheck can read, and read() loads that structured format into R as a paper object — everything Metacheck does, in every later chapter, starts from a paper object like this one.
Since our example papers are stored on GitHub, we first download the PDF with download.file(), a base R function that needs no extra packages:
Now we convert the PDF and read it in with the Metacheck convert function:
json_file <- convert(file_path = "good_paper.pdf", save_path = ".")
paper <- read(json_file)convert() sends the PDF to a free conversion server and gets back a structured file, which it saves in the folder named in save_path (here ".", the working directory; you must always name one). This needs an internet connection and takes a few seconds. You only need to do this once per paper — after that, you can read() the saved file directly, without converting again.
Note
convert() defaults to Metacheck’s own server on SURF Research Cloud in the Netherlands, which deletes your PDF once it is converted, and the result an hour later. See Reading in a Paper if you would rather run your own, fully local conversion server.
Tip
The good paper also comes with Metacheck, so you can try the steps below without downloading or converting anything: paper <- demopaper("stroop_effect") gives the paper object, and demofile("repo") gives the path to a copy of its repository on your computer.
3.2 Step 2: Run one check
Metacheck organises its checks into modules. Each module offers an automated check for one specific thing. module_run() runs a single module on a paper object. Just printing the result shows a short report:
module_run(paper, "stat_effect_size")Effect Sizes in t-tests and F-tests: All detected t-tests and F-tests had an effect size reported in the same sentence.
stat_effect_size checks whether a reported statistical test (like a t-test) is accompanied by an effect size, and whether that effect size is consistent with the test. Every module gives a traffic light — green (no issues found), yellow (worth a closer look), red (a likely problem), or “na” (nothing for this module to check) — plus a short summary. The good paper reports t(130) = 18.04 together with Cohen’s dz = 1.58, and that effect size is indeed what you would expect from a test statistic that size — hence the green light above.
Now download the bad paper’s PDF the same way as in Step 1, and run the same check on it:
download.file(
"https://raw.githubusercontent.com/Lakens/stroop_effect_paper_bad/main/analysis.pdf",
destfile = "bad_paper.pdf", mode = "wb"
)
json_file <- convert(file_path = "bad_paper.pdf", save_path = ".")
bad_paper <- read(json_file)module_run(bad_paper, "stat_effect_size")Effect Sizes in t-tests and F-tests: We found 1 t-test and/or F-test where effect sizes are not reported. Check these tests in the table below, and consider adding effect sizes.
This comes back as a red traffic light, which indicates something is wrong: it reports the same t-test, t(130) = 18.04, but with a Cohen’s dz of 0.25 — a value nowhere near what that test statistic implies. Metacheck does not know which number was typed wrong, but it correctly flags that the two do not agree, which is exactly the kind of inconsistency a reader re-typing numbers by hand is likely to miss.
3.2.1 Getting specific pieces of information out
Printing the result is useful for reading, but if you want to use a piece of it elsewhere — say, to check many papers automatically — assign the result to a variable first, and then pull out just the part you need with $:
mo_bad <- module_run(bad_paper, "stat_effect_size")
mo_bad$traffic_light#> [1] "red"mo_bad$summary_text#> [1] "We found 1 t-test and/or F-test where effect sizes are not reported. Check these tests in the table below, and consider adding effect sizes."mo_bad$traffic_light gives you just the colour, and mo_bad$summary_text gives you just the plain-language summary — the same two pieces of text that were part of the printed report above. Every module’s result has more than this; see Running the module in the Modules part of this book for what else is in there.
You now know the single most important function in Metacheck: module_run(paper, "module_name"). Every module in the Modules part of this book works exactly this way — only the module name changes.
3.3 Step 3: Create a full report
Most of the time you will not want to run modules one at a time — you want every relevant check, in one document. report() runs a whole battery of modules and writes the result to an HTML file you can open in a browser:
report(paper, output_file = "good_paper_report.html")This writes good_paper_report.html into your working directory. Open it, and you will see the same traffic lights and summaries from Step 2, but now for every module the report includes — preregistration, funding, conflicts of interest, power, statistical reporting, and more — all in one place, ready to read or share.
You already have the bad paper read in from Step 2 as bad_paper, so running the same check on it just means replacing paper with bad_paper:
report(bad_paper, output_file = "bad_paper_report.html")This produces the same kind of report, but with several more red and yellow lights: a missing funding statement, p-values that do not match their own reported statistics, and more. Nothing about the process changes between the two papers — only what the report finds.
3.5 What you have just done
In four short steps, you:
- Read a PDF into R as a paper object (
convert()+read()). - Ran a single check on it (
module_run()). - Generated a full report from a paper (
report()). - Generated a full report from a shared repository, with no manuscript involved (
report_repository()).
These four functions cover the large majority of what you will do with Metacheck day to day if you want to improve the reporting practices in your own work. Metacheck can also be run on a large collection of papers, and generate a dataframe that contains the results across the set of papers. This metascience use-case is explain in a dedicated section in each chapter of a module. The rest of this book goes through every module in detail — one chapter per module — documenting everything the R package can do with it: every argument, every option, what the output table and report actually contain, and how to interpret what it finds. Currently, these chapters are largely written by AI, with some light editing. If the Metacheck team has more time, and the software is more stable, we will revisit all chapters and manually update them. Start with the Modules part of this book, or jump straight to a chapter that interests you.