Report from module output
Arguments
- module_output
the output of a
module_run()- header
header level (default 2)
Examples
paper <- demopaper()
op <- module_run(paper, "stat_p_exact")
module_report(op) |> cat()
#> ### ⚠️ Exact P-Values {#exact-p-values .red}
#>
#> We found 1 imprecise *p* value out of 3 detected *p* values.
#>
#> <details><summary>View detailed feedback</summary>
#>
#> Reporting *p* values imprecisely (e.g., *p* < .05) reduces transparency, reproducibility, and re-use (e.g., in *p* value meta-analyses). Best practice is to report exact p-values with three decimal places (e.g., *p* = .032) unless *p* values are smaller than 0.001, in which case you can use *p* < .001.
#>
#> This module sometimes flags false positives when sentences discuss how p-value thresholds were set. It can also miss an imprecise p-value that is reported against a threshold (e.g., "decreased at the *p* < .05 significance threshold").
#>
#>
#> ```{r}
#> #| echo: false
#>
#>
#> # table data --------------------------------------
#> table <- structure(list("P-Value" = "p > .05", Text = "There was no effect of experience on the reduction in errors when using the tool (p > .05), as the correlation was non-significant (Figure 2)."), row.names = c(NA,
#> -1L), class = c("tbl_df", "tbl", "data.frame"))
#>
#> # display table -----------------------------------
#> metacheck::report_table(table, c(0.1, 0.9), 2, FALSE)
#> ```
#>
#>
#> ::: {.callout-tip title="Learn More" collapse="true"}
#>
#> The APA manual states: Report exact *p* values (e.g., *p* = .031) to two or three decimal places. However, report *p* values less than .001 as *p* < .001. However, 2 decimals is too imprecise for many use-cases (e.g., a *p* value meta-analysis), so report *p* values with three digits.
#>
#> American Psychological Association (2020). <em>Publication manual of the American Psychological Association</em>, 7 edition. American Psychological Association.
#>
#> :::
#>
#>
#> </details>
#>
#> ::: {.callout-note title="How It Works" collapse="true"}
#>
#> List any p-values reported with insufficient precision (e.g., p < .05 or p = n.s.) or reported as exactly zero (e.g., p = .000).
#>
#> This module uses regular expressions to identify p-values. It will flag any values reported as p > ? or p < numbers greater than .001. It will also flag p-values reported as exactly zero (e.g., p = .000, p = 0.00), which are mathematically impossible — p-values are never exactly zero and should instead be reported as p < .001.
#>
#> A p-value that is not the result of a test is not flagged: a table note ("* p < .05, ** p < .01"), a figure legend ("Asterisks indicate significant differences (p < .05)", "Values in bold are significant at p < .05") or a stated threshold ("p < .05 was considered significant", "thresholded at p < .001"). GROBID often puts such notes into the running text, so each p-value is judged by the words in its own clause (the text up to a ";" or a full stop), and never left out when a test statistic belongs to it ("F(1, 7) = 8.03, # p < .05"). The column `not_a_test` of the table says why a p-value was left out ("marker", "threshold" or "legend"). Notes in other forms, such as a marker after the p-value ("p < .05 (*)"), are still flagged.
#>
#> This module sometimes flags false positives when sentences discuss how p-value thresholds were set. It can also miss an imprecise p-value that is reported against a threshold ("decreased at the p < .05 significance threshold").
#>
#> This module only checks p-values reported in the running text of the manuscript. It cannot (yet) process p-values reported only in tables (as opposed to a table's footnote text, which the module does see and tries to exclude — see above).
#>
#> This module was developed by Lisa DeBruine and Daniel Lakens
#>
#> Read more in the [metacheck manual](https://www.scienceverse.org/metacheck_book/chapters/mod-stat-p-exact.html).
#>
#> :::
#>
#>
#> ::: {.validation}
#> Validation: In a sample of 225 papers containing 405 instances of non-exact p-values, the module correctly detected 269 cases (true positives) and incorrectly identified 78 (false positives). It missed 136 instances of imprecisely reported p-values (false negatives) and correctly identified 4557 cases of precisely reported p-values (true negative). Additionally, 78% of positive detections were correct (positive predictive value).
#> :::