A power analysis is a calculation researchers do, usually before collecting data, to work out how many participants they need to have a good chance of detecting an effect if it is really there. power searches a paper’s text for sentences that describe a power analysis.
10.2 What it checks
The power module uses regular expressions to find sentences that describe a statistical power analysis, and classifies each as a priori, sensitivity, compromise, or post-hoc (or as unknown when the sentence does not say which). When LLM support is turned on, it additionally reads each power sentence and extracts structured details — the statistical test, planned sample size, alpha level, desired power, and effect size — and checks whether the analysis is fully reported.
A priori power analyses are the useful kind; post-hoc (“observed”) power is rarely informative.
10.3 Running the module
NoteWithout R
This module also runs on Metacheck’s website: upload a paper under Run Metacheck on a Paper, where it is one of the default checks under Checks. The Docker installation offers the same website on your own computer. The website runs the module with its default settings; to change them, run it in R as below. On the website, the power check uses the server’s language model when the server has one; in the Docker installation, the model you chose when installing.
Every module is run the same way, with module_run(paper, "module_name"). We use the built-in demopaper(), which contains a few power statements.
paper<-demopaper()module_run(paper, "power")
Power Analysis Check: We detected 3 potential power analyses.
Printing the result renders the module’s report. To work with the underlying data, use the returned list. The table has one row per detected power statement, with a power_type classification and a complete column. Without an LLM enabled — the default, and what runs below — complete is always NA: it only becomes a real flag once LLM support is turned on, which is covered later in this chapter.
This paper demonstrates some good and poor practices for use with the {metacheck} R package. All data are simulated. The paper shows examples of (1) open and closed OSF links; (2a) citation of retracted papers, (2b) citations without a doi, (2c) citations with Pubpeer comments, (2d) citations in the FLoRA replication database, and (2e) missing/mismatched/incorrect citations and references; (3a) R files with code on GitHub that do not load libraries in one location, (3b) load files that are not shared in the repository, (3c) lack comments, and (3d) have absolute file paths; (4) imprecise reporting of non-significant p-values; (5) tests with and without effect sizes; (6) use of “marginally significant” to describe non-significant findings; (7) a power analysis reporting some of the essential attributes; and (8) retrieving information from preregistrations.
unknown
NA
We conducted a sensitivity power analysis 2 to determine that a Cohen’s d of 0.50 is the smallest effect size that we could detect with N = 50 participants in each group and 80% power.
sensitivity
NA
pwr::pwr.t.test(n = 50, power = 0.8, alternative = “greater”) is useful. We conclude the use of automated checks has potential to reduce the number of mistakes in scientific manuscripts.
unknown
NA
10.4 Running on many papers
Pass a paperlist such as the psychsci corpus (250 open-access Psychological Science articles, loaded with papers_load("psychsci", cache = TRUE) — see Using the Paper Database Corpora) to run across a corpus. Here we run on the first 20 papers and summarise how many power statements were found per paper.
A paper that reports an a priori power analysis with all the needed details is the “good” case; a paper that mentions power only vaguely, or reports post-hoc power, is the kind of thing the module surfaces for a closer look.
# demopaper contains both a priori and less-complete power statementsmodule_run(demopaper(), "power")
Power Analysis Check: We detected 3 potential power analyses.
The power_type column tells you which statements are a priori (worth keeping) versus post-hoc (usually a red flag). complete flags a priori analyses that are missing required information, but — as noted above — only once an LLM is enabled; without one it stays NA for every row, as in the example above.
10.6 Options: using an LLM
The regular-expression behaviour above runs with no setup. To also extract structured details — the statistical test, sample size, alpha, power, and effect size — turn on LLM support. The module reads the same switch (llm_use()) and model (llm_model()) described in the LLM chapter; there is no special module argument. When an LLM is enabled, the result table gains columns such as statistical_test, sample_size, power, effect_size, and effect_size_metric.
NoteThe language model on the website and in the Docker installation
On Metacheck’s website on SURF Research Cloud, the power check uses a small open-source Qwen model (qwen3.5:4b) that runs on the server itself. This can be slow, especially when several people use the server at the same time. That is a deliberate choice: the model runs on SURF’s servers in the Netherlands, so the paragraphs are never sent to an external AI provider, and the server stays GDPR compliant. In the Docker installation, the installer can set up the same model in Ollama on your own computer, so the paragraphs stay there, or use Mistral (EU) with your own API key.
Below we run exactly the same call three ways — with no LLM, with a local Ollama model, and with a cloud model on Groq — on demopaper(), and compare what each returns. (The LLM outputs shown here were captured from real runs; running the LLM at build time would require Ollama or an API key, so the results are loaded from saved objects.)
WarningCaptured with an earlier version of the module
The Ollama and Groq outputs below were captured in June 2026, with an earlier version of the power module. Since then the module has changed: it asks the model for structured answers, matches the model’s answers to known values (see How the model’s answers are matched to known values below), and leaves out a row for which the model gives no type of power analysis. Running the same calls today can therefore give other rows and other values than the tables below.
10.6.1 No LLM (default)
llm_use(FALSE)module_run(demopaper(), "power")
This is the regex-only behaviour shown earlier: it detects and classifies power statements, but the extraction columns are empty.
10.6.2 Local LLM with Ollama
This run used qwen3.5:9b (a 9.7-billion-parameter model) running locally with Ollama. Nothing leaves your machine, but the model does the work, so it is slower (this call took about 105 seconds).
The local model found the one genuine power analysis in the demo paper — the sensitivity analysis — and correctly extracted Cohen’s d = 0.5 and 80% power. It did not flag anything else.
10.6.3 Cloud LLM with Groq
This run used Groq’s default model (llama-3.1-8b-instant). It is much faster (about 11 seconds) but sends the text to an external service.
In these runs with the earlier version of the module, the two models behaved noticeably differently on the same input:
Recall vs. precision. The local qwen3.5:9b returned a single, clean row — the real sensitivity analysis. Groq’s smaller, faster llama-3.1-8b-instant returned three rows: it caught the same sensitivity analysis, but also flagged the paper’s abstract (a false positive, with every field NA) and a line of pwr.t.test() example code (extracting software = R and statistical_test = t-test). The current version leaves out a row without a type of power analysis, so an empty row like the one for the abstract no longer appears, and it matches answers such as R and t-test to known values, keeping an answer that matches none as "other" with the model’s own words in software_other or statistical_test_other.
Extraction detail. Both correctly pulled Cohen’s d = 0.5 and power = 0.8 for the sensitivity analysis. Neither marked any analysis as complete, because the demo paper does not report every required field.
Speed vs. privacy. Groq returned in ~11 s versus ~105 s locally, but the local run kept the manuscript on the machine. For unpublished work, the slower local model is usually the right trade-off.
There is no single “correct” answer here — the output depends on the model. A larger or more capable model generally gives cleaner extraction with fewer false positives, which is exactly why the model you choose matters. This is also why module output is meant to prompt human judgement rather than serve as a verdict: a person reading these tables quickly sees that the abstract and the code line are not real power analyses. See the Local AI with Ollama chapter for more on choosing and running local models.
10.6.5 How the model’s answers are matched to known values
The model answers four questions in its own words: the type of power analysis, the statistical test, the effect size measure, and the software. Metacheck then matches each answer to a list of known values, so power_type, statistical_test, effect_size_metric and software always hold one of those values. Earlier versions gave the model the fixed lists to choose from, but some providers (such as Groq) reject an answer outside the list, and one unexpected value then lost the whole power analysis. An answer that matches no known value becomes "other" (for the type of power analysis, "unknown"), and the model’s own words are kept in statistical_test_other, effect_size_metric_other or software_other. The software list holds the power-analysis and sample-size packages on CRAN, widely used packages whose title does not mention power (such as MBESS, faux and simglm), packages no longer on CRAN or from Bioconductor, and other programs such as G*Power, POWSIM, nQuery, GLIMMPSE, jamovi, JASP, SAS, Stata and SPSS.
10.6.6 Paragraphs the model could not check
When the request for a paragraph fails, for example because the provider did not answer, that paragraph is not counted as having no power analysis. The summary text says how many paragraphs could not be checked, the traffic light is yellow instead of green or “not applicable” (a red light stays red), and the summary table has a power_unchecked column with the number of such paragraphs per paper, so a run over many papers can find those papers and check them again. When no paragraph at all could be checked, the summary text says that the check failed to run, which is not the same as finding no power analyses, and the traffic light is “not applicable”.
Metacheck needs ellmer 0.4.2 or later for this module: with older versions, Groq rejects every structured request.
10.6.7 Reproducibility
The power module accepts a seed argument (default 8675309) passed to the LLM, so repeated runs with the same model are more reproducible:
module_run(demopaper(), "power", seed =1)
10.7 Validation
In a sample of 128 papers with 246 instances of power statements, 203 were correctly detected (true positives), 22 were missed (false negatives) and 21 were incorrectly detected (false positives). Among all instances flagged as power statements, 90.6% were correct (positive predictive value).
As with all modules, treat the output as a prompt for human judgement, not a verdict.