The Metacheck Manual

Automated Checks of Research Outputs for Best Practices

A guide to Metacheck, an R package providing a module based tool for checking scientific manuscripts and data repositories for best practices using text search, R code, and optional LLM queries.

Welcome

This is the online book for metacheck, an R package that provides extendable and integrated tools for automated checks of research outputs for best practices.

Inspired by practices in software development — where automated checks (e.g., CRAN checks for R packages) identify issues before release — Metacheck screens scientific manuscripts to flag potential issues or areas for improvement, and guides researchers in adopting best practices. It can also assist with processing large numbers of papers for metascientific enquiry.

These checks are meant to assist, not replace, human expertise. Metacheck can identify incorrect citations, add missing DOIs, cross-reference citations against Retraction Watch, PubPeer, and the FLoRA replication database, and flag commonly misreported aspects such as power analyses. Actioning this advice is always up to a human who can interpret it in context.

How this book is organised

• Getting Started — a conceptual overview of Metacheck and how to set it up, reading papers in, searching their text, using large language models, the paper database corpora, and writing your own modules.
• Modules — a chapter for each built-in module that the Shiny app runs, with worked examples on real papers: power analyses, preregistration retrieval and comparison (RegCheck), ethics approval, marginal significance, statistical reporting checks, repository and code checks, and reference checks against PubPeer, Retraction Watch, and the replication database.
• Using Metacheck Locally — running everything on your own machine without sending data anywhere: checking local files, and running AI models (and RegCheck) locally with Ollama.
• Appendices — reference material: the OSF registration schemas and a function reference for new users.

The values that guide all Scienceverse projects, including Metacheck, are described in the Introduction.

You can contact the Metacheck team at metacheck.scienceverse@gmail.com.