Metacheck is designed modularly, so you can add modules to check for anything. It comes with a set of pre-defined modules, and we hope people will share more modules.
Module List
You can see the list of built-in modules with the function below.
*** GENERAL ***
- all_urls: List all the URLs in the main text.
- coi_check: Identify and extract Conflicts of Interest (COI) statements.
- coi_check_oi: Identify and extract Conflicts of Interest (COI) statements.
- funding_check: Identify and extract funding statements.
- funding_check_oi: Identify and extract funding statements.
- open_practices: This module searches for open data, code, materials, and registration statements.
*** METHOD ***
- causal_claims: Aims to identify the presence of random assignment, and lists sentences that make causal claims in title or abstract.
- power: This module uses uses regular expressions to identify sentences that contain a statistical power analysis. If specified by the user, it also uses a large language module (LLM) to extract information reported in power analyses, including the statistical test, sample size, alpha level, desired level of power, and magnitude and type of effect size.
- prereg_check: Retrieve information from preregistrations in a standardised way, and make them easier to check.
*** RESULTS ***
- all_p_values: List all p-values in the text, returning the matched text (e.g., ‘p = 0.04’) and document location in a table.
- code_check: This module retrieves information from repositories checked by repo_check about code files (R, SAS, SPSS, Stata).
- marginal: List all sentences that describe an effect as ‘marginally significant’.
- repo_check: This module retrieves information from repositories.
- stat_check: Check consistency of p-values and test statistics
- stat_effect_size: The Effect Size module checks for effect sizes in t-tests and F-tests.
- stat_p_exact: List any p-values reported with insufficient precision (e.g., p < .05 or p = n.s.) or reported as exactly zero (e.g., p = .000).
- stat_p_nonsig: This module checks for imprecisely reported p values. If p > .05 is detected, it warns for misinterpretations.
*** REFERENCE ***
- ref_accuracy: This module checks references for mismatches with CrossRef.
- ref_consistency: Check if all references are cited and all citations are referenced
- ref_miscitation: Check for frequently miscited papers. This module is just a proof of concept – the miscite database is not yet populated with real examples.
- ref_pubpeer: This module checks references and warns for citations that have comments on pubpeer (excluding Statcheck comments).
- ref_replication: This module checks references and warns for citations of original studies for which replication or reproduction studies exist in the FLoRA database.
- ref_retraction: This module checks references and warns for citations in the RetractionWatch Database.
- ref_summary: Summarise information about each reference in a paper.
Use module_help("module_name") for help with a specific
module
Module Output
Module designers can include any information in the returned output, but we suggest they structure it in a specific way to facilitate creating reports and summarising many papers in a metascientific workflow.
So most modules output a list with the following named items: module,
title, table, report, traffic_light, summary_text, summary_table, paper.
You probably don’t need to worry about any of this unless you are
designing modules or using metacheck for metascience – the
report() function takes care of displaying everything for
you when you need to assess a single paper.
paper <- demopaper()
mo <- module_run(paper, "stat_p_exact")The module, title, and
summary_text give brief information.
mo$module#> [1] "stat_p_exact"
mo$title#> [1] "Exact P-Values"
mo$summary_text#> [1] "We found 1 imprecise *p* value out of 3 detected *p* values."Traffic light
The traffic_light helps the reports give a quick visual
guide to where there are problems or things to check.
mo$traffic_light#> [1] "red"🟢 no problems detected;
🟡 something to check;
🔴 possible problems detected;
🔵 informational only;
⚪️ not applicable;
⚫️ check failed
Table
The table is usually a detailed table in the format
returned from text_search() or text_expand(),
containing either text relevant to the module, or a classification of
the text. This table can be of use to further modules in a chain, or to
metascientific users.
mo$table#> # A tibble: 3 × 14
#> text text_id paragraph_id section_id page_number formatted paper_id header
#> <chr> <int> <int> <int> <int> <chr> <chr> <chr>
#> 1 p = 0.0… 15 4 3 NA NA to_err_… Proce…
#> 2 p =0.152 16 5 3 NA NA to_err_… Proce…
#> 3 p > .05 17 6 3 NA "There w… to_err_… Proce…
#> # ℹ 6 more variables: section_type <chr>, p_comp <chr>, p_value <dbl>,
#> # expanded <chr>, imprecise <lgl>, zero <lgl>Summary Table
The summary_table contains a single row for each paper,
and must have an id column that matches the paper IDs. It
will also have additional columns that summarise the results of the
module. This is mainly useful in the metascientific workflow, and this
table is appended by each module in a chain.
mo$summary_table#> paper_id n_imprecise n_zero
#> 1 to_err_is_human 1 0Report
The report contains a vector of markdown and R code to
be inserted into a report. The display is usually handled by the
module_report() function inside the report()
function.
mo$report#> [1] "Reporting *p* values imprecisely (e.g., *p* < .05) reduces transparency, reproducibility, and re-use (e.g., in *p* value meta-analyses). Best practice is to report exact p-values with three decimal places (e.g., *p* = .032) unless *p* values are smaller than 0.001, in which case you can use *p* < .001."
#> [2] "\n```{r}\n#| echo: false\n\n\n# table data --------------------------------------\ntable <- structure(list(\"P-Value\" = \"p > .05\", Text = \"There was no effect of experience on the reduction in errors when using the tool (p > .05), as the correlation was non-significant (Figure 2).\"), row.names = c(NA, \n-1L), class = c(\"tbl_df\", \"tbl\", \"data.frame\"))\n\n# display table -----------------------------------\nmetacheck::report_table(table, c(0.1, 0.9), 2, FALSE)\n```\n"
#> [3] "::: {.callout-tip title=\"Learn More\" collapse=\"true\"}\n\nThe APA manual states: Report exact *p* values (e.g., *p* = .031) to two or three decimal places. However, report *p* values less than .001 as *p* < .001. However, 2 decimals is too imprecise for many use-cases (e.g., a *p* value meta-analysis), so report *p* values with three digits.\n\nAmerican Psychological Association (2020). <em>Publication manual of the American Psychological Association</em>, 7 edition. American Psychological Association.\n\n:::\n"Paper
The paper is just the paper argument to
module_run(). This is mainly used when chaining
modules.
mo$paper#> ---------------
#> to_err_is_human
#> ---------------
#>
#> To Err is Human: An Empirical Investigation
#>
#> * Sections: 14
#> * Sentences: 37
#> * Bibliography: 5
#> * X-Refs: 5Previous Outputs
If you run modules in a chain or via the report()
function, the output accumulates the outputs of previous modules in this
item. This is so some modules can share resource-intensive parts of
checks rather than repeating them.
mo <- paper |>
module_run("stat_p_exact") |>
module_run("marginal") |>
module_run("stat_effect_size")
mo$prev_outputs#> $stat_p_exact
#> Exact P-Values: We found 1 imprecise *p* value out of 3 detected *p* values.
#> $marginal
#> Marginal Significance: You described 2 effects with terms related to 'marginally significant'.Built-in Modules
Below, we will demonstrate the use of a few built-in modules, first
on a single paper and then a list of papers, the psychsci
list of 250 open-access papers from Psychological Science.
paper <- psychsci$`0956797620955209`all_p_values
List all p-values in the text, returning the matched text (e.g., ‘p = 0.04’) and document location in a table.
all_p <- module_run(paper, "all_p_values")
all_p$table # print table#> # A tibble: 20 × 11
#> text text_id paragraph_id section_id page_number formatted paper_id header
#> <chr> <int> <int> <int> <int> <chr> <chr> <chr>
#> 1 "p = .… 63 15 6 NA NA 0956797… Quest…
#> 2 "p = .… 63 15 6 NA NA 0956797… Quest…
#> 3 "p < .… 101 24 9 NA "We obse… 0956797… Resul…
#> 4 "p < .… 102 24 9 NA "The mai… 0956797… Resul…
#> 5 "p = .… 102 24 9 NA "The mai… 0956797… Resul…
#> 6 "p = .… 102 24 9 NA "The mai… 0956797… Resul…
#> 7 "p = .… 103 24 9 NA NA 0956797… Resul…
#> 8 "p = .… 103 24 9 NA NA 0956797… Resul…
#> 9 "p = .… 104 24 9 NA "We also… 0956797… Resul…
#> 10 "p =.3… 104 24 9 NA "We also… 0956797… Resul…
#> 11 "p = .… 107 25 9 NA "Yes-the… 0956797… Resul…
#> 12 "p <.0… 107 25 9 NA "Yes-the… 0956797… Resul…
#> 13 "p = .… 109 26 9 NA NA 0956797… Resul…
#> 14 "p = .… 109 26 9 NA NA 0956797… Resul…
#> 15 "p = .… 109 26 9 NA NA 0956797… Resul…
#> 16 "p = .… 109 26 9 NA NA 0956797… Resul…
#> 17 "p < .… 110 26 9 NA "Results… 0956797… Resul…
#> 18 "p = .… 110 26 9 NA "Results… 0956797… Resul…
#> 19 "p = .… 110 26 9 NA "Results… 0956797… Resul…
#> 20 "p = .… 110 26 9 NA "Results… 0956797… Resul…
#> # ℹ 3 more variables: section_type <chr>, p_comp <chr>, p_value <dbl>If you run this module on all 250 papers, you will get more rows than you probably want to print in the full table one row for every p-value in each paper), so you can print the summary table, which gives you one row per paper.
all_p_ps <- module_run(psychsci, "all_p_values")
all_p_ps$summary_table |> head()#> paper_id p_values
#> 1 0956797613520608 6
#> 2 0956797614522816 39
#> 3 0956797614527830 13
#> 4 0956797614557697 24
#> 5 0956797614560771 4
#> 6 0956797614566469 0You can still access the full table for further processing.
#> # A tibble: 6 × 2
#> text n
#> <chr> <int>
#> 1 p < .001 1485
#> 2 p < .01 138
#> 3 p < .05 129
#> 4 p = .001 119
#> 5 p = .002 93
#> 6 p < .0001 88all_urls
List all the URLs in the main text. There will, of course, be a few false positives when text in the paper is formatted as a valid URL.
all_urls <- module_run(paper, "all_urls")
all_urls$table#> # A tibble: 6 × 9
#> text text_id paragraph_id section_id page_number formatted paper_id header
#> <chr> <int> <int> <int> <int> <chr> <chr> <chr>
#> 1 3.9.1.7 39 9 3 NA "Accordi… 0956797… Parti…
#> 2 https:/… 80 19 8 NA "Analysi… 0956797… Analy…
#> 3 https:/… 130 30 13 NA "All dat… 0956797… Open …
#> 4 https:/… 131 31 13 NA "The des… 0956797… Open …
#> 5 http://… 133 31 13 NA "More in… 0956797… Open …
#> 6 http://… 138 34 16 NA "Additio… 0956797… Suppl…
#> # ℹ 1 more variable: section_type <chr>
all_urls_ps <- module_run(psychsci, "all_urls")
all_urls_ps$summary_table#> paper_id urls
#> 1 0956797613520608 1
#> 2 0956797614522816 0
#> 3 0956797614527830 1
#> 4 0956797614557697 6
#> 5 0956797614560771 0
#> 6 0956797614566469 5
#> 7 0956797615569001 9
#> 8 0956797615569889 3
#> 9 0956797615583071 4
#> 10 0956797615588467 2
#> 11 0956797615603702 0
#> 12 0956797615615584 2
#> 13 0956797615617779 1
#> 14 0956797615620784 4
#> 15 0956797615625973 5
#> 16 0956797616631990 6
#> 17 0956797616634654 2
#> 18 0956797616634665 1
#> 19 0956797616636631 6
#> 20 0956797616647519 8
#> 21 0956797616657319 4
#> 22 0956797616661199 5
#> 23 0956797616663878 5
#> 24 0956797616665351 7
#> 25 0956797616667447 1
#> 26 0956797616669994 1
#> 27 0956797616671327 2
#> 28 0956797616671712 2
#> 29 0956797617692000 7
#> 30 0956797617693326 1
#> 31 0956797617694867 9
#> 32 0956797617702501 6
#> 33 0956797617702699 5
#> 34 0956797617705391 4
#> 35 0956797617705667 4
#> 36 0956797617707270 7
#> 37 0956797617710785 6
#> 38 0956797617714811 1
#> 39 0956797617716922 3
#> 40 0956797617716929 10
#> 41 0956797617724435 9
#> 42 0956797617736886 10
#> 43 0956797617737129 31
#> 44 0956797617739368 9
#> 45 0956797617740685 3
#> 46 0956797617744542 19
#> 47 0956797618755322 7
#> 48 0956797618760197 4
#> 49 0956797618772822 4
#> 50 0956797618773095 1
#> 51 0956797618785899 8
#> 52 0956797618795679 3
#> 53 0956797618796480 6
#> 54 0956797618804501 2
#> 55 0956797618815482 0
#> 56 0956797618815488 1
#> 57 0956797618823540 2
#> 58 0956797619830326 20
#> 59 0956797619830329 11
#> 60 0956797619831964 4
#> 61 0956797619833325 2
#> 62 0956797619835147 9
#> 63 0956797619837981 1
#> 64 0956797619841265 8
#> 65 0956797619842261 7
#> 66 0956797619842550 6
#> 67 0956797619844231 7
#> 68 0956797619851753 4
#> 69 0956797619866625 6
#> 70 0956797619866627 8
#> 71 0956797619869905 5
#> 72 0956797619876260 10
#> 73 0956797619881134 7
#> 74 0956797619890619 7
#> 75 0956797620903716 21
#> 76 0956797620904450 4
#> 77 0956797620904990 18
#> 78 0956797620915887 21
#> 79 0956797620916521 3
#> 80 0956797620916782 14
#> 81 0956797620927648 5
#> 82 0956797620927967 8
#> 83 0956797620929297 2
#> 84 0956797620929302 7
#> 85 0956797620931108 4
#> 86 0956797620939054 15
#> 87 0956797620941840 6
#> 88 0956797620948821 20
#> 89 0956797620951115 7
#> 90 0956797620954815 1
#> 91 0956797620955209 6
#> 92 0956797620957625 9
#> 93 0956797620958638 2
#> 94 0956797620958650 3
#> 95 0956797620959014 2
#> 96 0956797620959594 19
#> 97 0956797620960011 7
#> 98 0956797620963615 11
#> 99 0956797620965520 4
#> 100 0956797620965536 6
#> 101 0956797620967261 5
#> 102 0956797620968789 2
#> 103 0956797620970548 5
#> 104 0956797620970559 3
#> 105 0956797620971298 8
#> 106 0956797620971652 4
#> 107 0956797620972116 4
#> 108 0956797620972688 2
#> 109 0956797620975781 6
#> 110 0956797620984464 5
#> 111 0956797620985832 9
#> 112 09567976211001317 6
#> 113 09567976211005465 3
#> 114 09567976211005767 7
#> 115 09567976211007414 15
#> 116 09567976211007788 15
#> 117 09567976211010718 10
#> 118 09567976211011969 18
#> 119 09567976211013045 1
#> 120 09567976211015941 5
#> 121 09567976211015942 6
#> 122 09567976211016395 1
#> 123 09567976211016410 9
#> 124 09567976211017870 8
#> 125 09567976211018618 17
#> 126 09567976211019950 11
#> 127 09567976211024259 17
#> 128 09567976211024260 4
#> 129 09567976211024535 10
#> 130 09567976211026983 5
#> 131 09567976211028978 5
#> 132 09567976211030630 4
#> 133 09567976211032224 6
#> 134 09567976211032676 5
#> 135 09567976211037971 9
#> 136 09567976211040491 14
#> 137 09567976211040803 11
#> 138 09567976211043426 6
#> 139 09567976211043428 4
#> 140 09567976211046884 3
#> 141 09567976211048485 1
#> 142 09567976211049439 16
#> 143 09567976211051272 9
#> 144 09567976211052476 7
#> 145 09567976211055375 13
#> 146 09567976211059801 5
#> 147 09567976211061321 14
#> 148 09567976211068045 3
#> 149 09567976211068070 3
#> 150 09567976211068880 3
#> 151 0956797621991137 3
#> 152 0956797621991548 6
#> 153 0956797621995197 11
#> 154 0956797621995202 16
#> 155 0956797621996660 7
#> 156 0956797621996667 8
#> 157 0956797621997350 4
#> 158 0956797621997366 11
#> 159 0956797621998312 5
#> 160 09567976221079633 2
#> 161 09567976221082637 7
#> 162 09567976221082938 11
#> 163 09567976221082941 11
#> 164 09567976221083219 10
#> 165 09567976221086513 4
#> 166 09567976221089599 7
#> 167 09567976221094036 13
#> 168 09567976221094782 9
#> 169 09567976221101045 8
#> 170 09567976221114055 8
#> 171 09567976221116816 2
#> 172 09567976221116892 7
#> 173 09567976221116893 6
#> 174 09567976221119391 7
#> 175 09567976221121348 5
#> 176 09567976221131519 5
#> 177 09567976221131520 9
#> 178 09567976221134476 7
#> 179 09567976221139496 1
#> 180 09567976221140326 1
#> 181 09567976221140341 16
#> 182 09567976221145316 5
#> 183 09567976221147258 2
#> 184 09567976221147259 6
#> 185 09567976221150616 5
#> 186 09567976231151581 2
#> 187 09567976231154804 3
#> 188 09567976231156413 6
#> 189 09567976231156793 1
#> 190 09567976231158288 6
#> 191 09567976231158570 3
#> 192 09567976231160098 1
#> 193 09567976231160702 10
#> 194 09567976231161565 5
#> 195 09567976231164553 3
#> 196 09567976231165267 1
#> 197 09567976231170878 2
#> 198 09567976231172500 3
#> 199 09567976231173900 17
#> 200 09567976231173902 3
#> 201 09567976231177968 2
#> 202 09567976231179378 2
#> 203 09567976231180578 10
#> 204 09567976231180588 3
#> 205 09567976231180881 5
#> 206 09567976231184887 8
#> 207 09567976231185127 3
#> 208 09567976231185129 8
#> 209 09567976231188107 12
#> 210 09567976231188124 0
#> 211 09567976231190546 7
#> 212 09567976231192241 1
#> 213 09567976231194221 1
#> 214 09567976231194590 5
#> 215 09567976231196145 6
#> 216 09567976231198194 11
#> 217 09567976231198435 4
#> 218 09567976231199440 4
#> 219 09567976231203139 1
#> 220 09567976231204035 12
#> 221 09567976231207095 8
#> 222 09567976231213572 10
#> 223 09567976231217508 1
#> 224 09567976231218640 2
#> 225 09567976231220902 7
#> 226 09567976231221789 2
#> 227 09567976231222288 4
#> 228 09567976231222836 4
#> 229 09567976231223130 7
#> 230 09567976231223410 10
#> 231 09567976241227411 7
#> 232 09567976241228504 4
#> 233 09567976241232891 4
#> 234 09567976241235931 4
#> 235 09567976241235932 0
#> 236 09567976241239932 4
#> 237 09567976241239935 9
#> 238 09567976241242105 5
#> 239 09567976241243370 4
#> 240 09567976241245695 10
#> 241 09567976241246561 3
#> 242 09567976241249183 5
#> 243 09567976241254312 4
#> 244 09567976241258149 9
#> 245 09567976241260247 3
#> 246 09567976241263344 9
#> 247 09567976241263347 7
#> 248 09567976241266516 5
#> 249 09567976241267854 5
#> 250 09567976241279291 11stat_p_exact
List any p-values that may have been reported with insufficient precision (e.g., p < .05 or p = n.s.).
imprecise <- module_run(paper, "stat_p_exact")
imprecise$table$text # print table#> [1] "p = .003" "p = .08" "p < .001 " "p < .025" "p = .040" "p = .173"
#> [7] "p = .006" "p = .02" "p = .691" "p =.303" "p = .023" "p <.001"
#> [13] "p = .006" "p = .037" "p = .038" "p = .358" "p < .001" "p = .127"
#> [19] "p = .062" "p = .047"The expanded column has the full sentence for context.
Here you can see that “p < .025” was not an imprecisely reported
p-value, but a description of the preregistered alpha threshold.
imprecise$table$expanded[[4]] # print expanded text#> [1] "The main effect of illness recency did not meet our preregistered threshold (p < .025)-recently ill: M = 661 ms, SD = 197; not recently ill: M = 626 ms, SD = 153, F(1, 400) = 4.23, ηp² = .010, 90% CI = [.000, .039], p = .040-nor did the interaction between illness recency and face type (disfigured vs. typical), F(1, 400) = 1.87, ηp² = .005, 90% CI = [.000, .027], p = .173."We can investigate the most common imprecise p-values in the PsychSci set. “p < .01” and “p < .05” are probably often describing figures or tables, but what is the deal with “p > .25”?
imprecise_ps <- module_run(psychsci, "stat_p_exact")
imprecise_ps$table |>
count(text, sort = TRUE) |>
head()#> # A tibble: 6 × 2
#> text n
#> <chr> <int>
#> 1 p < .001 1485
#> 2 p < .01 138
#> 3 p < .05 129
#> 4 p = .001 119
#> 5 p = .002 93
#> 6 p < .0001 88We can expand the text to check the context for “p > .25”.
gt.25 <- imprecise_ps$table |>
filter(grepl("\\.25", text))
gt.25$expanded[1:3] # look at the first 3#> [1] "There was a significant interactive effect of time and political orientation, b = 0.10, SE = 0.04, 95% CI = [0.03, 0.17], t(1922) = 2.72, p = .007, on endorsement of the fairness foundation (see Table S2 CI = [-0.14, -0.04], t(1922) = -3.59, p < .001, disappeared after 7/7, b = 0.004, SE = 0.02, 95% CI = [-0.04, 0.05], t(1922) = 0.17, p > .250 (see Fig 2)."
#> [2] "Contrary to expectations, our results revealed no significant main effect of time, b = -0.13, SE = 0.22, 95% CI = [-0.55, 0.30], t(1922) = -0.58, p > .250, political orientation, b = 0.05, SE = 0.10, 95% CI = [-0.15, 0.24], t(1922) = 0.47, p > .250, or their interaction, b = 0.04, SE = 0.06, 95% CI = [-0.08, 0.16], t(1922) = 0.67, p > .250, on endorsement of the authority foundation."
#> [3] "Contrary to expectations, our results revealed no significant main effect of time, b = -0.13, SE = 0.22, 95% CI = [-0.55, 0.30], t(1922) = -0.58, p > .250, political orientation, b = 0.05, SE = 0.10, 95% CI = [-0.15, 0.24], t(1922) = 0.47, p > .250, or their interaction, b = 0.04, SE = 0.06, 95% CI = [-0.08, 0.16], t(1922) = 0.67, p > .250, on endorsement of the authority foundation."marginal
List all sentences that describe an effect as ‘marginally significant’.
marginal <- module_run(paper, "marginal")
marginal # print tableMarginal Significance: You described 0 effects with terms related to ‘marginally significant’.
Let’s check how many are in the full set.
marginal_ps <- module_run(psychsci, "marginal")
marginal_ps$table # print table#> # A tibble: 102 × 9
#> text text_id paragraph_id section_id page_number formatted paper_id header
#> <chr> <int> <int> <int> <int> <chr> <chr> <chr>
#> 1 Althou… 128 29 15 NA "Althoug… 0956797… Detai…
#> 2 A marg… 139 32 18 NA "A margi… 0956797… Postt…
#> 3 When w… 150 35 19 NA NA 0956797… The e…
#> 4 In tha… 154 36 20 NA "In that… 0956797… Condi…
#> 5 The Co… 110 22 7 NA "The Con… 0956797… Resul…
#> 6 The tw… 113 24 6 NA "The two… 0956797… Resul…
#> 7 An omn… 186 41 14 NA NA 0956797… Resul…
#> 8 Furthe… 241 63 28 NA NA 0956797… NA
#> 9 The da… 255 65 30 NA "The dag… 0956797… Fig 5
#> 10 Given … 166 38 19 NA NA 0956797… Autho…
#> # ℹ 92 more rows
#> # ℹ 1 more variable: section_type <chr>stat_check
Check consistency of p-values and test statistics using functions from statcheck.
statcheck <- module_run(paper, "stat_check")
statcheck$table#> test_type df1 df2 test_comp test_value p_comp reported_p computed_p
#> 1 t NA 248.4 = 2.01 = 0.023 4.551244e-02
#> 2 t NA 248.4 = -4.55 < 0.001 8.397343e-06
#> raw error decision_error one_tailed_in_txt apa_factor
#> 1 t(248.4) = 2.01, p = .023 TRUE FALSE FALSE 1
#> 2 t(248.4) = -4.55, p <.001 FALSE FALSE FALSE 1
#> text
#> 1 Yes-the 90% confidence intervals of the difference in attentional bias for participants who were and were not recently ill found here (dz = -0.14, 90% CI = [-0.31, -0.04]) did not overlap with an effect size (dz ) of -0.35, t(248.4) = 2.01, p = .023, or 0.35, t(248.4) = -4.55, p <.001.
#> 2 Yes-the 90% confidence intervals of the difference in attentional bias for participants who were and were not recently ill found here (dz = -0.14, 90% CI = [-0.31, -0.04]) did not overlap with an effect size (dz ) of -0.35, t(248.4) = 2.01, p = .023, or 0.35, t(248.4) = -4.55, p <.001.
#> text_id paragraph_id section_id page_number
#> 1 107 25 9 NA
#> 2 107 25 9 NA
#> formatted
#> 1 Yes-the 90% confidence intervals of the difference in attentional bias for participants who were and were not recently ill found here (dz = -0.14, 90% CI = [-0.31, -0.04]) did not overlap with an effect size (dz ) of -0.35, t(248.4) = 2.01, p = .023, or 0.35, t(248.4) = -4.55, p <.001.
#> 2 Yes-the 90% confidence intervals of the difference in attentional bias for participants who were and were not recently ill found here (dz = -0.14, 90% CI = [-0.31, -0.04]) did not overlap with an effect size (dz ) of -0.35, t(248.4) = 2.01, p = .023, or 0.35, t(248.4) = -4.55, p <.001.
#> paper_id header section_type
#> 1 0956797620955209 Results results
#> 2 0956797620955209 Results resultsHere we see a false positive, where the paper reported the results of an equivalence test, which are meant to be one-tailed, but statcheck did not detect that this was one-tailed.
In the full PsychSci set, there are more than 27K sentences with numbers to check, so this takes about a minute to run.
statcheck_ps <- module_run(psychsci, "statcheck")There will be, of course, some false positives in the full set of 151 flagged values. Let’s look just at the flagged values where the computed p-value is about double the reported p-value, and this changes the significance decision (at an alpha of 0.05).
statcheck_ps$table |>
filter(decision_error,
round(computed_p/reported_p, 1) == 2.0) |>
select(reported_p, computed_p, raw) |>
mutate(computed_p = round(computed_p, 4))#> reported_p computed_p raw
#> 1 0.0290 0.0589 F(1, 361) = 3.59, p = .029
#> 2 0.0470 0.0947 t(24) = 1.74, p = .047
#> 3 0.0270 0.0547 t(24) = 2.02, p = .027
#> 4 0.0400 0.0797 t(24) = 1.83, p = .040
#> 5 0.0480 0.0962 t(240) = 1.67, p = .048
#> 6 0.0460 0.0915 t(32) = 1.74, p = .046
#> 7 0.0420 0.0846 t(21) = 1.81, p = .042
#> 8 0.0343 0.0686 t(10) = 2.04, p = .0343
#> 9 0.0330 0.0654 t(55) = 1.88, p = .033Chaining Modules
Modules return a summary table as well as the detailed
results table, which is automatically added to the summary
if you chain modules.
ps_metascience <- psychsci[1:10] |>
module_run("all_p_values") |>
module_run("stat_p_exact") |>
module_run("marginal")
ps_metascience$summary_table#> paper_id p_values n_imprecise n_zero marginal
#> 1 0956797613520608 6 0 0 0
#> 2 0956797614522816 39 0 0 0
#> 3 0956797614527830 13 2 0 0
#> 4 0956797614557697 24 8 0 0
#> 5 0956797614560771 4 1 0 0
#> 6 0956797614566469 0 0 0 0
#> 7 0956797615569001 25 20 0 0
#> 8 0956797615569889 28 0 0 4
#> 9 0956797615583071 25 2 0 0
#> 10 0956797615588467 21 4 0 0