Match reported statistical tests against the extracted analysis output

Recomposes the statistics a paper REPORTS (from paper$eq, grouped by the grp_id that ties one reported test's numbers together) into whole tests, then checks whether each test's component values co-occur in a single analysis of the paper's extracted OUTPUT (reproducibility_check's statistical output). A test is matched on its whole signature, not lone numbers, so a match means the reported result actually appears in the reproducible output rather than a value coinciding by chance. The result carries, per matched test, which output file and analysis produced it (provenance).

Usage

match_reported_output(
  paper,
  output,
  include_tables = FALSE,
  min_components = 1L
)

Arguments

paper: a paper object (its $eq is recomposed), or an eq data frame
output: the extracted output: a stat_results_long() data frame, or the stat_output list a reproducibility_check result carries (its per-file $long tables are combined, keeping each file's provenance)
include_tables: if TRUE and paper is a paper object, also build tests from statistics reported only in a results table's cells (e.g. from Grobid's table parsing – see .tei_table_contents() in R/import-grobid.R and the 5-tier header/caption/value-shape/row/matrix approach in R/match-table.R's .table_tests()) and add them to the set checked against output. Default FALSE: table-derived tests are opt-in, currently used only by reproducibility_check, so every other caller's matching behaviour is unchanged.
min_components: a test must have at least this many recognised components to be assessed (default 1, so a lone reported statistic – a bare correlation r=-.27 or d = 0.40 with no accompanying p/CI – is still checked against the output, TYPE-GATED the same as any other component: it only matches an output cell of the SAME family, at the site with the most co-occurring components, via val_in(). This carries a higher coincidence risk than a multi-component test (a single r rounding to the reported value at some site is more likely than a whole signature doing so by chance), so a single-component match is worth reading with that in mind – set to 2 to exclude single-component rows entirely, as this function used to by default.

Value

a data.frame, one row per recomposed reported test, with: text_id, grp_id, reported (the recomposed test as text, e.g. "W=183.5 p=.791 rb=-0.16"), n_components, n_matched (components found in the best-matching output analysis), found (logical), match_values (the matched components as "name=value" pairs), not_matched (the UNMATCHED components of that same test, same "name=value" form – always populated alongside match_values for a partial match, so exactly which claim failed to reproduce is visible, not just a count), source_file / analysis (provenance of the match), confidence ("full" all components matched / "partial" / "none"), and plausible_split – NA unless this test's grouping came from .regroup_by_evidence() re-deriving test boundaries from OUTPUT co-occurrence rather than trusting extract_tests()'s text-only guess (see that function's own header comment): TRUE when a component split away from its original reported neighbours landed at a site sharing a row_label token with theirs (the same underlying variable, reached via a different analysis – plausible), FALSE when no such link was found (the split is NOT hidden or downgraded, only flagged, so a reader can judge the match's plausibility from the source_file/analysis of every piece involved). Attribute "summary" holds the roll-up.

Examples

# run the paper's paired t-test and compare it with what the paper reports
stroop <- read.csv(file.path(demofile("repo"), "data", "stroop_data.csv"))
out <- utils::capture.output(
  t.test(stroop$incongruent, stroop$congruent, paired = TRUE))
long <- stat_results_long(read_r_output(out))
match_reported_output(demopaper("stroop_effect"), long)
#>   text_id grp_id                                  reported n_components
#> 1      17      1                             M=15.1 SD=4.1            2
#> 2      17      1                              M=23 SD=4.78            2
#> 3      18      3                                    r=0.37            1
#> 4      18      3                    t=4.53 df=129 p=<0.001            3
#> 5      19      5                   t=18.04 df=130 p=<0.001            3
#> 6      19      5                           Cohen's dz=1.58            1
#> 7      24      6                                    M=7.77            1
#> 8      24      6                                    M=8.13            1
#> 9      24      6 t=-0.42 df=120.37 p=0.673 Cohen's d=-0.07            4
#>   n_matched found              match_values not_matched source_file
#> 1         0 FALSE                      <NA>        <NA>        <NA>
#> 2         0 FALSE                      <NA>        <NA>        <NA>
#> 3         0 FALSE                      <NA>        <NA>        <NA>
#> 4         1 FALSE                      <NA>        <NA>        <NA>
#> 5         3  TRUE t=18.04, df=130, p=<0.001                    <NA>
#> 6         0 FALSE                      <NA>        <NA>        <NA>
#> 7         0 FALSE                      <NA>        <NA>        <NA>
#> 8         0 FALSE                      <NA>        <NA>        <NA>
#> 9         0 FALSE                      <NA>        <NA>        <NA>
#>        analysis confidence plausible_split
#> 1          <NA>       none            TRUE
#> 2          <NA>       none            TRUE
#> 3          <NA>       none            TRUE
#> 4          <NA>       none            TRUE
#> 5 Paired t-test       full           FALSE
#> 6          <NA>       none           FALSE
#> 7          <NA>       none            TRUE
#> 8          <NA>       none            TRUE
#> 9          <NA>       none            TRUE