19 Exact p-Values

19.1 What it checks

The stat_p_exact module flags two kinds of problematic p-values:

Imprecise p-values — for example p < .05 or p = n.s. instead of an exact value like p = .031.
P-values reported as exactly zero — for example p = .000 or p = 0.00. A p-value is never exactly zero; p = .000 is a rounding artifact (the true value is simply smaller than the reported precision) and should be written p < .001.

Each detected value carries an imprecise flag and a zero flag, and either kind turns the traffic light red. When a p = .000 is found, the report includes its own guidance explaining why exactly-zero p-values should be reported as p < .001.

Detection is case-insensitive, so an upper-case P = .04 is found just as reliably as p = .04.

A p-value that is not the result of a test is not flagged as imprecise:

a table note, such as “* p < .05, ** p < .01”, “ns: p > .05” or “† p < .10”;
a figure legend, such as “Asterisks indicate significant differences (p < .05)” or “Values in bold are significant at p < .05”;
a stated threshold, such as “p < .05 was considered significant”, “thresholded at P < .001” or “a Bonferroni-corrected alpha of p < .02”.

GROBID often puts table notes and figure legends into the running text, so the section a sentence is in cannot tell them apart from results. Each p-value is therefore judged by the words of its own clause (the text up to a “;” or a full stop), and is never left out when a test statistic belongs to it (“F(1, 7) = 8.03, # p < .05”). The column not_a_test of the table says why a p-value was left out ("marker", "threshold" or "legend"), and is NA for the p-values that were judged. Notes in other forms, such as a marker after the p-value (“p < .05 (*)”), are still flagged.

The words that mark a threshold cannot always tell a sentence that sets a threshold from one that reports a result against it. The module therefore sometimes flags false positives when sentences discuss how p-value thresholds were set, and it can miss an imprecise p-value that is reported against a threshold, such as “decreased at the p < .05 significance threshold”. The report says so when it lists imprecise p-values.

Reporting exact p-values is an APA Journal Article Reporting Standard (JARS). It also lets readers include results in p-curve or z-curve meta-analyses and check internal consistency with tools like Statcheck.

This module is fully offline.

19.2 Running the module

Without R

This module also runs on Metacheck’s website: upload a paper under Run Metacheck on a Paper, where it is one of the default checks under Checks. The Docker installation offers the same website on your own computer. The website runs the module with its default settings; to change them, run it in R as below.

paper <- demopaper()
module_run(paper, "stat_p_exact")

Exact P-Values: We found 1 imprecise p value out of 3 detected p values.

The table lists every detected p-value, with imprecise and zero flags, and in not_a_test the reason a p-value was left out:

mo <- module_run(paper, "stat_p_exact")
mo$traffic_light

#> [1] "red"

mo$summary_text

#> [1] "We found 1 imprecise *p* value out of 3 detected *p* values."

mo$table[, c("text", "p_comp", "p_value", "imprecise", "zero", "not_a_test")] |>
  knitr::kable()

text	p_comp	p_value	imprecise	zero	not_a_test
p = 0.005	=	0.005	FALSE	FALSE	NA
p =0.152	=	0.152	FALSE	FALSE	NA
p > .05	>	0.050	TRUE	FALSE	NA

To see only the problematic ones:

mo$table[mo$table$imprecise | mo$table$zero, c("text", "expanded")] |>
  knitr::kable()

text	expanded
p > .05	There was no effect of experience on the reduction in errors when using the tool (p > .05), as the correlation was non-significant (Figure 2).

19.3 Running on many papers

The summary_table reports a per-paper count for each problem type: n_imprecise (imprecise p-values) and n_zero (p-values reported as exactly zero).

mo <- module_run(psychsci[1:20], "stat_p_exact")
head(mo$summary_table) |>
  knitr::kable()

paper_id	n_imprecise	n_zero
0956797613520608	0	0
0956797614522816	5	0
0956797614527830	3	0
0956797614557697	7	0
0956797614560771	1	0
0956797614566469	0	0

The p-values in these papers that were left out because they are not the result of a test:

not_tests <- mo$table[!is.na(mo$table$not_a_test), c("paper_id", "expanded", "not_a_test")]
head(not_tests, 10) |>
  knitr::kable()

paper_id	expanded	not_a_test
0956797614527830	(Statistical significance was tested against a threshold of p < .05; bootstrapped models that provided 95% confidence intervals for the coefficients described below can be found in Table S2 in the Supplemental Material.)	threshold
0956797614527830	Asterisks indicate a significant difference between conditions ( * * * p < .005).	marker
0956797614557697	Asterisks indicate a significant difference between TMS conditions (p < .05).	legend
0956797614557697	Asterisks indicate a significant difference between conditions (p < .01).	legend
0956797615569001	Social Class I was the reference group. p < .05. *p <.01.	marker
0956797615569001	Social Class I was the reference group. p < .05. *p <.01.	marker
0956797615569001	Low self-control was defined as scoring 1 standard deviation below the mean on the standardized self-control measure, medium self-control was defined as scoring at the mean on this measure, and high self-control was defined as scoring at the maximum of 0.83 standard deviations above the mean. **p <.01.	marker
0956797615569889	Asterisks indicate significant differences between conditions (p < .05; p < .01; p < .001).	marker
0956797615569889	Asterisks indicate significant differences between conditions (p < .05; p < .01; p < .001).	marker
0956797615569889	Asterisks indicate significant differences between conditions (p < .05; p < .01; p < .001).	marker

19.4 A clean example and one with problems

A known paper that reports imprecise p-values:

module_run(psychsci$`0956797614560771`, "stat_p_exact")

Exact P-Values: We found 1 imprecise p value out of 4 detected p values.

A paper that reports a p-value as exactly zero (p = .000). This is flagged separately from imprecise values, with its own guidance that exactly-zero p-values should be reported as p < .001:

module_run(psychsci$`0956797615603702`, "stat_p_exact")

Exact P-Values: We found 1 p value reported as exactly zero out of 8 detected p values.

A paper that reports all p-values exactly returns green:

module_run(psychsci$`0956797616665351`, "stat_p_exact")

Exact P-Values: We found no imprecise p values or p-values of exactly zero out of 8 detected.

19.5 Options

stat_p_exact takes only the paper argument.

19.6 Validation

In a sample of 225 papers containing 405 instances of non-exact p-values, the module correctly detected 269 cases (true positives) and incorrectly identified 78 (false positives). It missed 136 instances (false negatives) and correctly identified 4557 cases of precisely reported p-values (true negatives). 78% of positive detections were correct (positive predictive value).