The marginal module lists every sentence that describes an effect as “marginally significant” (or a related phrase). Describing a non-significant result as “marginally significant”, “approaching significance”, or “trending toward significance” is a common way to over-interpret a result that did not pass the chosen threshold. The module surfaces these sentences so a reader can judge each in context.
The term list is based on Matthew Hankins’ well-known catalogue of phrases used to dress up non-significant results (see this blog post by Matthew Hankins).
This module is fully offline — it uses only regular expressions, so all the code below runs with just the Metacheck package.
15.2 Running the module
paper<-demopaper()module_run(paper, "marginal")
Marginal Significance: You described 2 effects with terms related to ‘marginally significant’.
The table contains one row per flagged sentence:
mo<-module_run(paper, "marginal")mo$traffic_light
#> [1] "red"
mo$summary_text
#> [1] "You described 2 effects with terms related to 'marginally significant'."
The paper shows examples of (1) open and closed OSF links; (2a) citation of retracted papers, (2b) citations without a doi, (2c) citations with Pubpeer comments, (2d) citations in the FLoRA replication database, and (2e) missing/mismatched/incorrect citations and references; (3a) R files with code on GitHub that do not load libraries in one location, (3b) load files that are not shared in the repository, (3c) lack comments, and (3d) have absolute file paths; (4) imprecise reporting of non-significant p-values; (5) tests with and without effect sizes; (6) use of “marginally significant” to describe non-significant findings; (7) a power analysis reporting some of the essential attributes; and (8) retrieving information from preregistrations.
abstract
On average researchers in the experimental condition found the app marginally significantly more useful (M = 5.06) than researchers in the control condition found the checklist (M = 4.5), t(97.2) = -1.96, p =0.152.
method
15.3 Running on many papers
mo<-module_run(psychsci[1:30], "marginal")# papers where at least one marginal-significance phrase was foundsubset(mo$summary_table, marginal>0)|>head()|>knitr::kable()
paper_id
marginal
8
0956797615569889
4
13
0956797615617779
1
18
0956797616634665
1
19
0956797616636631
1
29
0956797617692000
2
15.4 A clean example and one with problems
A paper with no such phrasing returns a green traffic light and an empty table:
clean<-test_paper("The effect was significant, t(28) = 2.40, p = .02, d = 0.45.")module_run(clean, "marginal")$traffic_light
#> [1] "green"
A paper that hedges a non-significant result is flagged:
flagged<-test_paper("The difference was marginally significant (p = .07).")mo<-module_run(flagged, "marginal")mo$traffic_light
#> [1] "red"
mo$table$text
#> [1] "The difference was marginally significant (p = .07)."
15.5 Options
marginal takes no options beyond the paper argument — it is a pure text search.
15.6 Validation
In a sample of 51 papers with 87 statements, this module correctly identified 38 statements (true positives) and incorrectly flagged 22 statements (false positives). It failed to detect 27 statements. Among all statements flagged by the module, 63% were genuine cases (positive predictive value); the module missed 42% of all true statements (false negative rate).
15.7 List of terms
The module looks for six families of phrasing, each describing a result as nearly significant. In every case the wording is allowed to appear a few words before “significant” (so “marginally but not quite significant” still matches). The families are:
Phrasing
Matches (examples)
margin… significant
marginally significant, marginal significance
trend… significant
trend toward significance, trending significant
almost … significant
almost significant, almost reached significance
approach… significant
approaching significance, approached significance
border… significant
borderline significant, bordering on significance
close to … significant
close to significant, close to being significant
Under the hood these are a single regular expression. You can always inspect the exact pattern the installed module uses with module_info():
module_info("marginal")
The regex itself is:
"margin\\w* (?:\\w+\\s+){0,5}significan\\w*|trend\\w* (?:\\w+\\s+){0,1}significan\\w*|almost (?:\\w+\\s+){0,2}significan\\w*|approach\\w* (?:\\w+\\s+){0,2}significan\\w*|border\\w* (?:\\w+\\s+){0,2}significan\\w*|close to (?:\\w+\\s+){0,2}significan\\w*"
As the validation note above shows, this catches many but not all ways of describing marginal significance, and some matches may be acceptable in context.
Because both the false-positive and false-negative rates are appreciable, this module is best used as a first-pass highlighter for human review.