18  Non-Significant p-Values

18.1 What it checks

The stat_p_nonsig module finds every non-significant p-value (p ≥ .05) so a reader can check whether the surrounding text interprets it correctly. A common error is to treat a non-significant result as evidence that there is no effect (“there was no difference between groups, p = .27”), when a non-significant result is simply inconclusive.

The module makes no judgement itself — it only surfaces the sentences so you can read how each result is described.

This module is fully offline.

18.2 Running the module

paper <- demopaper()
module_run(paper, "stat_p_nonsig")

Non-Significant P Value Check: We found 2 non-significant p values that should be checked for appropriate interpretation.

The table lists each non-significant p-value with its comparison and value:

mo <- module_run(paper, "stat_p_nonsig")
mo$traffic_light

#> [1] "yellow"

mo$summary_text

#> [1] "We found 2 non-significant p values that should be checked for appropriate interpretation."

mo$table[, c("text", "p_comp", "p_value", "significance")] |>
  knitr::kable()

text	p_comp	p_value	significance
p =0.152	=	0.152	nonsignificant
p > .05	>	0.050	nonsignificant

18.3 Running on many papers

mo <- module_run(psychsci[1:20], "stat_p_nonsig")
head(mo$summary_table) |>
  knitr::kable()

paper_id	n_nonsignificant
0956797613520608	0
0956797614522816	17
0956797614527830	0
0956797614557697	12
0956797614560771	0
0956797614566469	0

18.4 A clean example and one with problems

A result that is correctly described as inconclusive is fine; one that claims “no effect” from a non-significant test is the kind of thing to check. The module flags both, and you read the sentence:

flagged <- test_paper("There was no difference between the groups, t(40) = 1.10, p = .28.")
mo <- module_run(flagged, "stat_p_nonsig")
mo$table[, c("text", "p_value")] |>
  knitr::kable()

text	p_value
p = .28	0.28

A paper reporting only significant results returns nothing to check:

clean <- test_paper("The effect was significant, t(28) = 2.40, p = .02.")
module_run(clean, "stat_p_nonsig")$traffic_light

#> [1] "green"

18.5 Options

stat_p_nonsig takes only the paper argument.

18.6 Validation

In a sample of 194 papers with 1602 instances of non-significant p-values, this module correctly detected 1486 of them and incorrectly identified 153. 91% of detections were true instances (positive predictive value).