2 Setting up Metacheck

2.1 Ways to use Metacheck

Route	What you need	Best for
Metacheck’s website on SURF Research Cloud	A web browser; nothing to install.	Checking a paper or a repository now and then. The server’s capacity is shared by everyone, so it is not meant for many papers.
Docker installation	Docker on your own computer: a computer with 8 GB of memory for the lightest installation, or 16 GB to also convert PDFs on your own computer. No R.	The same website on your own computer: for checking many papers, for example in a metascience project, and, with PDFs converted on your own computer, for manuscripts you may not share.
Your own server on SURF Research Cloud	Access to SURF Research Cloud, which researchers at Dutch research institutions can get through their institution or a free NWO grant. Nothing to install on your computer.	The same website on a server of your own, with more powerful hardware than the shared server: for many papers, for a research group, or with bibr and a graphics card.
Metacheck in R	R and RStudio.	Everything Metacheck can do: all modules and their settings, your own modules, and analyses across many papers.

2.2 The easiest way to get started

This manual mostly explains how to use Metacheck from R. If you do not want to install anything, use Metacheck’s website on SURF Research Cloud instead. It runs on SURF’s servers in the Netherlands: PDFs are converted to text on the server itself (in the paper report app you can also choose another converter), and the one check that uses a language model, the power check, uses a small open-source Qwen model on the same server, so your paper is not sent to an external provider; this can be slow (see Large language models). The site is GDPR compliant.

The server offers these tools:

Tool	What you can do with it
Metacheck website	Upload a paper as a PDF or a bibr JSON file and receive a report. Or choose a folder of data and code, a .zip file, or a link to an online repository (GitHub, OSF, Zenodo, Dataverse, Dryad, and others), and receive a report on the data and code; the R and Python code can also be run, in a closed sandbox without internet access.
TU/e Data Manager Check	Compare a data package with TU/e’s archiving guidelines, the way TU/e’s data managers check it before a package is archived.
ResearchBox Check	Check the data and code in a ResearchBox.
Paper report app	Create a report from a PDF, choosing which modules to run and whether the power check uses the language model.
Web interface	Use the same tools from R, Python or the command line (see below).

Screenshot of the Metacheck website, with a box to run Metacheck on a paper and a box to run Metacheck on a repository — The Metacheck website on SURF Research Cloud

You receive the report as a web page in your browser, which you can save. The report starts with an explanation of Metacheck, a legend of the symbols for each check (a possible problem, something to check, no problem, information only), and a table of contents that lists every check with its result. Below that come a summary and the results of each check.

Screenshot of the top of a Metacheck report, with the paper's title, an introduction, the legend of symbols, and a table of contents listing the checks and their results — The top of a Metacheck report

The data manager check and the ResearchBox check use the modules that check shared data and code, such as Data Check and Code Check.

Every tool can also be used from your own code through the web interface. For example, from a terminal:

curl -F file=@paper.pdf https://metacheckserve.metacheck.src.surf-hosted.nl/api/report -o report.html

The documentation of the web interface lists all options.

The options of the server are more limited than running Metacheck from R: for example, you cannot set the parameters of a module. Uploaded files are deleted as soon as they have been read, and reports are kept on the server for one hour, so save a report you want to keep. For everything else, install Metacheck in R (see below).

You can also run the same website on your own computer, without installing R, with the Docker installation; see Docker installation below. Please do so if you want to check many papers.

2.3 Installing in R

Before you install Metacheck, you need two things on your computer:

R, version 4.3.0 or newer. Download it for your operating system from https://cran.r-project.org/ and install it with the default options.
RStudio Desktop (the free edition), from https://posit.co/download/rstudio-desktop/. RStudio is not strictly required, but we assume it throughout this book. Recent versions of RStudio also bundle the Quarto command-line tool, which Metacheck uses to generate reports.

With R and RStudio installed, the recommended way to install Metacheck is from the scienceverse R-universe. R-universe provides ready-to-use, pre-compiled versions of Metacheck for Windows and macOS, so you do not need any additional build tools, and installation is a single command. Open RStudio and run:

install.packages("metacheck",
  repos = c("https://scienceverse.r-universe.dev", "https://cloud.r-project.org"))

This tells R to look for Metacheck on the scienceverse R-universe first, and to pull any other required packages from the regular CRAN mirror. On Windows and macOS this installs binary versions of every package, so nothing is built from source and the installation works on a fresh machine without any extra setup. The download includes Metacheck’s dependencies, so on a first installation it may take a few minutes.

After installation, you can load the Metacheck library.

library(metacheck)

Metacheck also has an app that opens in your web browser: you upload a PDF and get a report with one click. The app needs three packages that are not installed with Metacheck: shiny, shinydashboard and shinyjs. Install them once, and then start the app:

install.packages(c("shiny", "shinydashboard", "shinyjs"))
report_app()

Like report(), the app needs Quarto to write the report.

2.3.1 Installing the development version

Metacheck is under active development. The development version often has new modules or functions before they reach the stable release. R-universe only ever hosts one build per package name per universe, so the development version is not available as a pre-compiled binary; install it directly from GitHub instead. Installing this way builds Metacheck from source, which requires a compiler toolchain that does not ship with R by default:

Windows: Install Rtools, matching the version to your R version (Rtools44 for R 4.4.x, Rtools43 for R 4.3.x, and so on), from https://cran.r-project.org/bin/windows/Rtools/. Use the default options. This is a large download.
macOS: Install the Xcode command-line tools by running xcode-select --install in the Terminal. For some packages you may also need a Fortran compiler, available from https://mac.r-project.org/tools/.
Linux: Install the standard development tools (for example build-essential and gfortran on Debian/Ubuntu), along with the development versions of system libraries such as libcurl, libxml2, and libssl.

With the build tools in place, install the pak package and then install Metacheck from the dev branch:

install.packages("pak")
pak::pkg_install("scienceverse/metacheck@dev")

To install a different branch, change the name after the @. A source install keeps the package name metacheck, so you load it as usual with library(metacheck).

library(metacheck)

To check your first paper in R, continue with Your First Check.

2.3.2 Starting from a manuscript, or from a folder

report() (see Creating a Metacheck Report) starts from a manuscript: you read a paper, and the modules follow the repository links it contains. Sometimes there is no manuscript at all — you just have a folder of files, downloaded from a repository yourself or sent to you directly for review. For that case, use report_repository() instead: give it a path, and it runs the repository-oriented modules (repo_check, code_check, data_check, codebook_check) directly on the folder and writes a report named after it.

report_repository("path/to/folder")

See Checking Local Files for worked examples, including downloading a repository’s files yourself before running this.

2.3.3 Running locally for privacy

Metacheck has been developed so that it can run locally, without sending any information anywhere. This requires a bit more technical expertise, but it makes the tool available to editors or peer reviewers who are not allowed to upload submitted manuscripts to an external server.

When such privacy concerns are not an issue (for example when running Metacheck on a publicly available preprint) you can use online servers for specific tasks, such as converting a pdf to xml. This offers an easier user experience.

By default, convert() sends a PDF to Metacheck’s own server on SURF Research Cloud in the Netherlands, which converts it with bibr when bibr is available, and otherwise with GROBID on the same server. That server needs no API key, deletes the uploaded PDF once it is converted, and deletes the result an hour later. Only when that server cannot be reached does Metacheck try the GROBID server of Eindhoven University of Technology, then a GROBID server on your own computer, and then the public GROBID servers on Hugging Face from a list of active servers.

If you cannot send manuscripts to an external server, the easiest way to run all of Metacheck on your own computer is the Docker installation, with GROBID on your computer. To use Metacheck from R with your own GROBID server, start GROBID with Docker (the official image, version 0.9.1, without deep-learning models; it needs about 4 GB of memory); see https://grobid.readthedocs.io/ for other ways to run it:

docker run --rm --init --ulimit core=0 -p 8070:8070 grobid/grobid:0.9.1-crf

A running local GROBID server is not chosen automatically while the SURF server or the GROBID server of Eindhoven University of Technology answers, so to keep the manuscript on your computer, name the local server when you convert: convert(pdf_file, save_path = "converted", method = "grobid", api_url = "http://localhost:8070"). The same principle applies to large language models: the recommended setup runs a model locally so no data is sent to an external service (see Using Large Language Models).

See the Reading in a Paper chapter for the full details of converting and reading papers, including how to point convert() at a local server.

2.4 Docker installation

The Docker installation runs Metacheck on your own computer, without installing R. It contains the same tools as the server on SURF Research Cloud: the website (with the ResearchBox and TU/e Data Manager checks), the paper report app, and the web interface. One installer asks which parts to add. The lightest installation needs little memory, but sends some things to external servers; each part you add does more on your own computer instead.

Use your own installation for large projects. The capacity of the hosted servers is limited and shared by everyone who uses them. They are meant for checking a paper or a repository now and then. If you want to check many papers, for example in a metascience project, run Metacheck on your own computer with the Docker image (or from R), instead of sending the papers to the hosted servers.

2.4.1 What runs where

Part	Without it	With it
Metacheck (always)		The website, the paper report app, the web interface, and running the code of a repository, on your computer.
GROBID (converts PDFs)	A PDF is sent to the GDPR compliant GROBID server of Eindhoven University of Technology to be converted, or, when that server does not answer, to Metacheck’s server on SURF Research Cloud in the Netherlands, which deletes it after the conversion.	PDFs are converted on your computer.
Language model (for the power check)	The power check works without a language model.	Ollama runs the model on your computer, so the paragraphs stay there. Or, instead of Ollama, Mistral (EU) with your own API key: the paragraphs about statistical power are then sent to Mistral.

In every installation, the checks that look something up online (Crossref, PubPeer, OSF, GitHub, and so on) send what they look up, such as DOIs or repository links, and only when you choose those checks. When Metacheck starts, it says where PDFs are converted and which language model the power check uses.

2.4.2 What your computer needs

	Lightest installation	With GROBID
Memory	A computer with 8 GB of memory. Metacheck used 0.3 GB while waiting for work and at most 1.6 GB while making two reports at the same time, which fits in the 4 GB that Docker gets on such a computer (we have not yet tried one).	A computer with 16 GB of memory: GROBID needs another 3.7 GB.
Processor	A 64-bit Intel or AMD processor, or an Apple Silicon Mac; 4 cores are recommended (two reports at the same time used about 2.5 cores).	The same.
Disk space	About 1.7 GB for Metacheck, plus the space Docker Desktop itself needs.	About 1.7 GB more.
Software	Docker Desktop (Windows, macOS) or Docker Engine (Linux), which the installer can install. Docker Desktop is free for personal use, education, non-commercial open source projects, and small businesses; larger companies need a paid subscription.	The same.

On Windows, Docker Desktop gets half of the computer’s memory by default; on a Mac, check its memory setting in Docker Desktop under Settings > Resources. The language model is optional. With Ollama, the model of the power check (qwen3.5:4b) needs 3.3 GB of disk space; it ran entirely on a graphics card with 6 GB of memory, and without a suitable graphics card it runs on the processor, which is slower. With Mistral, you need an API key and an internet connection.

2.4.3 Installing

On Windows, open PowerShell and run:

irm https://raw.githubusercontent.com/scienceverse/metacheck_platform_docker/main/install.ps1 | iex

On macOS or Linux, open a terminal and run:

curl -fsSL https://raw.githubusercontent.com/scienceverse/metacheck_platform_docker/main/install.sh | bash

The installer asks what to install. Pressing Enter answers “no” each time, so without other answers you get the lightest installation.

Docker. If Docker is not installed, the installer offers to install it: Docker Desktop on Windows (which asks for administrator rights) and on macOS (with Homebrew), and Docker Engine on Linux. Metacheck cannot run without Docker. On Windows, Docker Desktop may ask you to restart the computer; run the installer again afterwards.
GROBID on this computer. Answer yes to convert PDFs on your computer, if it has 16 GB of memory.
Ollama. Answer yes to run the language model on your computer. The installer installs Ollama, sets it up as on the SURF server, and downloads the model.
A Mistral API key, when you did not choose Ollama. Mistral offers API keys on a free plan that needs no credit card: log in at console.mistral.ai, choose API Keys, and Create new key. Check Mistral’s terms for whether it may use what you send to improve its models. Paste the key, and the installer checks it with Mistral; or press Enter to skip this, and add a key later (see below).

The installer puts everything in a folder metacheck in your user folder, starts Metacheck, and opens http://localhost:3838 in your browser. The paper report app is at http://localhost:3838/metacheck/. Metacheck can only be reached from your own computer, not from other computers on your network. The folder holds the files to start and stop Metacheck: start-metacheck.cmd and stop-metacheck.cmd on Windows, start-metacheck.command and stop-metacheck.command on macOS (double-click them), and start-metacheck.sh and stop-metacheck.sh on Linux. Reports are kept for one hour while Metacheck runs, and are deleted when it stops, so save the reports you want to keep. The installer for macOS and Linux has not yet been tested on those systems.

2.4.4 Changing the settings later

The installer writes your choices to the file .env in the metacheck folder. To add a Mistral API key later, open that file in a text editor, change the line # MISTRAL_API_KEY= into MISTRAL_API_KEY= followed by your key, and stop and start Metacheck. You can also run the installer again, and answer the questions differently. Mistral is only used when Ollama is not set up. To never send a PDF to the SURF server when TU/e’s GROBID server does not answer, add the line METACHECK_CONVERT_BACKUP= to the file.

2.4.5 Running the code of a repository

The website can run the R and Python code of a repository, and compare its results with the paper. In the Docker installation, the code runs inside the Docker container; this needs no extra installation. The R and Python packages the code needs and Metacheck does not include are downloaded while the check runs, so it needs an internet connection and takes longer when a repository uses many packages. Unlike the closed sandbox on the SURF server, the container is not closed off: the code can use the internet. It cannot reach the files on your computer.

2.5 Your own server on SURF Research Cloud

Metacheck’s website on SURF Research Cloud is not only a shared server: the software that builds it is a building block of SURF Research Cloud, so you can run your own copy of it there. Your own server can have more processors and memory than the shared one, and a graphics card, which bibr needs and which makes the language model much faster.

2.5.1 What SURF Research Cloud is

SURF is the cooperative of the Dutch universities, universities of applied sciences, university medical centres and research institutes, and provides their national computing infrastructure. SURF Research Cloud is a portal where you build a virtual research environment yourself. The idea behind it:

You build the environment you need, when you need it. You choose a ready-made catalog item from a catalogue, choose the size of the computer (a flavour: the number of processors, the memory, and optionally a graphics card), and the portal creates a workspace: a virtual computer in SURF’s data centres in the Netherlands, with the software already installed.
The environment is described by scripts. A catalog item is a set of human-readable scripts that build the environment the same way each time. So you can delete a workspace when you no longer need it and build an identical one later, and anyone can see exactly what was installed.
You pay only for what runs. A workspace uses computing time (counted in processor hours, or graphics card hours) only while it runs; a paused workspace uses none, and keeps its configuration and data. A workspace itself is temporary: what should outlive it, such as files you want to keep, goes on a separate storage volume.
You work together in a collaboration. Access is arranged through SURF Research Access Management (SRAM): you log in with your institution’s account, and you can give colleagues access to your workspaces, also at other institutions and in other countries.

2.5.2 Metacheck’s building blocks

The scripts that build Metacheck’s server are open source, in scienceverse/surf-research-cloud. They provide two components, which can be combined in a catalog item:

metacheck: the website, the web interface and the paper report app, exactly as on Metacheck’s shared server, optionally with GROBID, LibreOffice for Word files, and a small language model on the workspace itself.
bibr: bibr, Scienceverse’s own pipeline for extracting the text, structure and references of scientific papers, and an alternative to GROBID. It converts PDF and Word files into the bibr JSON that Metacheck reads, in more detail than GROBID, and serves a language model that a metacheck workspace can use too. It needs a workspace with a graphics card (an A10) and a storage volume of at least 150 GB. A metacheck workspace in the same collaboration reaches it over SURF’s private network.

The repository’s README describes the catalog items (“metacheck server”, “bibr server”, and “metacheck + bibr” on one workspace), every setting, and how to add the components to your own collaboration’s catalogue.

2.5.3 Getting started

Get a budget. Check with your institution’s IT or research support whether it already has direct access to SURF’s compute services, through a contract with SURF. If not, apply for a free NWO small compute grant for SURF Research Cloud: it is meant for small projects and pilots, is granted within about two weeks, and lasts up to a year; it pays for SURF’s own HPC Cloud, the computers Research Cloud runs on in the Netherlands. The overview of access routes explains the options.
Log in to the SURF Research Cloud portal with your institution’s account. Your budget appears there as a wallet, which workspaces are paid from; if you have none, request one with the portal’s “Request” button and the number of your contract or grant. You work in a collaboration, with the colleagues you invite.
Get Metacheck’s catalog item. If the Metacheck team offers its catalog item “on request”, a member of your collaboration with developer rights can request it in the portal. Otherwise, a developer can build a catalog item from Metacheck’s components, following the README. The Metacheck team can help: metacheck@scienceverse.org.
Create the workspace. Choose the catalog item and a flavour; the portal shows how much each flavour costs per day. As an indication: Metacheck’s shared server runs on 4 processors and 32 GB of memory, which costs 96 processor hours a day, and a workspace with an A10 graphics card for bibr costs 24 graphics card hours a day. Attach a storage volume, so the use counts and reports outlive the workspace; you can attach it when you create the workspace, or later while the workspace is paused. Set the expiry date: on that date SURF deletes the workspace (after an e-mail three days before), and you can change it while the workspace runs.
Open your website at the address the portal shows, of the form https://<workspace>.<collaboration>.src.surf-hosted.nl/.

More information:

SURF Research Cloud, SURF’s page about the service, with a two-minute video
SURF’s documentation of Research Cloud, from first steps (getting a budget, starting a workspace) to building catalog items
SURF’s rates (in Dutch)
Introduction to SURF Research Cloud, SURF’s training, on 10 December 2026

2.6 Large language models

A few modules (notably power, data_check, and codebook_check) can use a large language model to extract structured information from text. LLM use is entirely optional and opt-in, and the vast majority of modules work without any LLM. For how to turn LLM support on, choose a model, run one locally with Ollama, or use a cloud API, see Using Large Language Models.

You can also use a language model without R. On Metacheck’s website on SURF Research Cloud, the power check uses a small open-source Qwen model (qwen3.5:4b) that runs on the server itself. This can be slow, especially when several people use the server at the same time. That is a deliberate choice: the model runs on SURF’s servers in the Netherlands, so the paragraphs are never sent to an external AI provider, and the server stays GDPR compliant. In the Docker installation, the installer can set up the same model in Ollama on your own computer, so the paragraphs stay there, or use Mistral (EU) with your own API key. On the website and in the Docker installation, only the power check uses the language model.

2.7 API keys and tokens for online repositories

The repo_check module and related functions can read files directly from repositories such as OSF, Zenodo, Dataverse, Dryad, Figshare, 4TU.ResearchData, and GitLab. For most of these an access token is optional and only raises how many requests you may make per hour, or gives you access to your own private material — the one exception is Dryad, where a token is required even to download files from a fully public dataset.

Each service issues its own token, which you store once in a file called .Renviron so R finds it automatically every time it starts. Step-by-step instructions, with screenshots, exist for OSF and Zenodo; for Dataverse, Dryad, Figshare, 4TU.ResearchData, and GitLab, see Getting API Tokens for Other Repositories. LLM provider API keys (OpenAI, Anthropic, Groq, and so on) are a separate matter, covered in Using Large Language Models.