17 Exact p-Values

17.1 What it checks

The stat_p_exact module flags p-values reported with insufficient precision — for example p < .05 or p = n.s. instead of an exact value like p = .031 — and p-values reported as exactly zero (p = .000), which is mathematically impossible and should be written p < .001.

Reporting exact p-values is an APA Journal Article Reporting Standard (JARS). It also lets readers include results in p-curve or z-curve meta-analyses and check internal consistency with tools like Statcheck.

This module is fully offline.

17.2 Running the module

paper <- demopaper()
module_run(paper, "stat_p_exact")

Exact P-Values: We found 1 imprecise p value out of 3 detected p values.

The table lists every detected p-value, with imprecise and zero flags:

mo <- module_run(paper, "stat_p_exact")
mo$traffic_light

#> [1] "red"

mo$summary_text

#> [1] "We found 1 imprecise *p* value out of 3 detected *p* values."

mo$table[, c("text", "p_comp", "p_value", "imprecise", "zero")] |>
  knitr::kable()

text	p_comp	p_value	imprecise	zero
p = 0.005	=	0.005	FALSE	FALSE
p =0.152	=	0.152	FALSE	FALSE
p > .05	>	0.050	TRUE	FALSE

To see only the problematic ones:

mo$table[mo$table$imprecise | mo$table$zero, c("text", "expanded")] |>
  knitr::kable()

text	expanded
p > .05	There was no effect of experience on the reduction in errors when using the tool (p > .05), as the correlation was non-significant (Figure 2).

17.3 Running on many papers

mo <- module_run(psychsci[1:20], "stat_p_exact")
head(mo$summary_table) |>
  knitr::kable()

paper_id	n_imprecise	n_zero
0956797613520608	0	0
0956797614522816	0	0
0956797614527830	2	0
0956797614557697	8	0
0956797614560771	1	0
0956797614566469	0	0

17.4 A clean example and one with problems

A known paper that reports imprecise p-values:

module_run(psychsci$`0956797614560771`, "stat_p_exact")

Exact P-Values: We found 1 imprecise p value out of 4 detected p values.

A paper that reports all p-values exactly returns green:

module_run(psychsci$`0956797616665351`, "stat_p_exact")

Exact P-Values: We found no imprecise p values or p-values of exactly zero out of 8 detected.

17.5 Options

stat_p_exact takes only the paper argument.

17.6 Validation

In a sample of 225 papers containing 405 instances of non-exact p-values, the module correctly detected 269 cases (true positives) and incorrectly identified 78 (false positives). It missed 136 instances (false negatives) and correctly identified 4557 cases of precisely reported p-values (true negatives). 78% of positive detections were correct (positive predictive value).