18 Exact p-Values

18.1 What it checks

The stat_p_exact module flags two kinds of problematic p-values:

Imprecise p-values — for example p < .05 or p = n.s. instead of an exact value like p = .031.
P-values reported as exactly zero — for example p = .000 or p = 0.00. A p-value is never exactly zero; p = .000 is a rounding artifact (the true value is simply smaller than the reported precision) and should be written p < .001.

Each detected value carries an imprecise flag and a zero flag, and either kind turns the traffic light red. When a p = .000 is found, the report includes its own guidance explaining why exactly-zero p-values should be reported as p < .001.

Reporting exact p-values is an APA Journal Article Reporting Standard (JARS). It also lets readers include results in p-curve or z-curve meta-analyses and check internal consistency with tools like Statcheck.

This module is fully offline.

18.2 Running the module

paper <- demopaper()
module_run(paper, "stat_p_exact")

Exact P-Values: We found 1 imprecise p value out of 3 detected p values.

The table lists every detected p-value, with imprecise and zero flags:

mo <- module_run(paper, "stat_p_exact")
mo$traffic_light

#> [1] "red"

mo$summary_text

#> [1] "We found 1 imprecise *p* value out of 3 detected *p* values."

mo$table[, c("text", "p_comp", "p_value", "imprecise", "zero")] |>
  knitr::kable()

text	p_comp	p_value	imprecise	zero
p = 0.005	=	0.005	FALSE	FALSE
p =0.152	=	0.152	FALSE	FALSE
p > .05	>	0.050	TRUE	FALSE

To see only the problematic ones:

mo$table[mo$table$imprecise | mo$table$zero, c("text", "expanded")] |>
  knitr::kable()

text	expanded
p > .05	There was no effect of experience on the reduction in errors when using the tool (p > .05), as the correlation was non-significant (Figure 2).

18.3 Running on many papers

The summary_table reports a per-paper count for each problem type: n_imprecise (imprecise p-values) and n_zero (p-values reported as exactly zero).

mo <- module_run(psychsci[1:20], "stat_p_exact")
head(mo$summary_table) |>
  knitr::kable()

paper_id	n_imprecise	n_zero
0956797613520608	0	0
0956797614522816	0	0
0956797614527830	2	0
0956797614557697	8	0
0956797614560771	1	0
0956797614566469	0	0

18.4 A clean example and one with problems

A known paper that reports imprecise p-values:

module_run(psychsci$`0956797614560771`, "stat_p_exact")

Exact P-Values: We found 1 imprecise p value out of 4 detected p values.

A paper that reports a p-value as exactly zero (p = .000). This is flagged separately from imprecise values, with its own guidance that exactly-zero p-values should be reported as p < .001:

module_run(psychsci$`0956797615603702`, "stat_p_exact")

Exact P-Values: We found 1 p value reported as exactly zero out of 8 detected p values.

A paper that reports all p-values exactly returns green:

module_run(psychsci$`0956797616665351`, "stat_p_exact")

Exact P-Values: We found no imprecise p values or p-values of exactly zero out of 8 detected.

18.5 Options

stat_p_exact takes only the paper argument.

18.6 Validation

In a sample of 225 papers containing 405 instances of non-exact p-values, the module correctly detected 269 cases (true positives) and incorrectly identified 78 (false positives). It missed 136 instances (false negatives) and correctly identified 4557 cases of precisely reported p-values (true negatives). 78% of positive detections were correct (positive predictive value).