29  Reference Consistency

Important

This module is not currently useful. It depends on the cross-references and bibliography that GROBID extracts from the PDF, and that extraction is too low in quality for the check to be reliable: GROBID frequently fails to link in-text citations to their reference, and mis-parses parts of the reference list into spurious entries. In practice the overwhelming majority of the discrepancies this module reports are artefacts of that extraction rather than real inconsistencies in the paper.

We are building a more accurate tool to convert PDFs to text, which should make this check dependable. Until then, treat its output as a measure of extraction quality rather than as a list of problems with the paper.

29.1 What it checks

The ref_consistency module checks the bibliography and the in-text citations against each other, in both directions:

  • Extra references — entries in the reference list that are never cited anywhere in the text.
  • Missing references — in-text citations that do not point to any entry in the reference list.

A paper where every reference is cited and every citation resolves to a reference is internally consistent; anything else is flagged.

29.2 Running the module

paper <- demopaper()
mo <- module_run(paper, "ref_consistency")
mo$traffic_light
#> [1] "red"
mo$summary_text
#> [1] "There are cross-references that are not in the bibliography and/or bibliography entries not cross-referenced in the text"

The summary_table gives per-paper counts: the number of bibliography entries (n_bib), the number of in-text cross-references (n_xrefs), and how many were flagged as n_extra (uncited references) or n_missing (citations with no matching reference):

mo$summary_table |>
  knitr::kable()
paper_id n_bib n_xrefs n_extra
to_err_is_human 5 0 5

The table has one row per flagged discrepancy. A row with contents = NA is an extra reference (a bibliography entry with no matching citation); a row with a citation in contents but no reference is a missing reference:

mo$table[, c("bib_id", "reference", "contents")] |>
  knitr::kable()
bib_id reference contents
0 DeBruine, Lisa. 2025. Faux: Simulation for Factorial Designs. Zenodo. https: //doi.org/10.5281/10.5281/zenodo.2669586. NA
1 Eagly, Alice H., and Wendy Wood. 1999. “The Origins of Sex Differences in Human Behavior: Evolved Dispositions Versus Social Roles.” American Psychologist 54 (6): 408-23. https://doi.org/10.1037/0003-066x.54.6.408. NA
2 Gino, Francesca, and Scott S. Wiltermuth. 2014. “Evil Genius? How Dishonesty Can Lead to Greater Creativity.” Psychological Science 25 (4): 973-81. https://doi.org/10.1177/0956797614520714. NA
3 Lakens, Daniël. 2018. “Equivalence Testing for Psychological Research.” Ad- vances in Methods and Practices in Psychological Science 1: 259-70. NA
4 Smith, F. 2021. “Human Error Is a Symptom of a Poor Design.” Journal of Journals. https://doi.org/10.0000/0123456789. NA

29.3 Running on many papers

mo <- module_run(psychsci[1:10], "ref_consistency")
mo$summary_table

29.4 Interpreting the result

As the callout above warns, a flag here is much more likely to reflect imperfect PDF extraction than a real problem with the paper. The two most common failure modes are:

  • Unlinked citations. GROBID detects an in-text citation but cannot attach it to a reference, so the citation is reported as missing even though the work is in the reference list. These dominate the “missing” flags.
  • Mis-parsed references. GROBID splits a single reference across rows, or turns a running head, page number, or DOI fragment into its own bibliography “entry”. These appear as extra references that were never really in the paper.

A genuinely uncited reference, or a citation to a work that is truly absent from the reference list, does occur — but it is the exception, and you can only confirm it by looking at the original PDF. Once the more accurate extraction tool is in place, this section will be revisited.

29.5 Options

ref_consistency takes only the paper argument.

29.6 Notes

This module sits alongside the other reference-section checks: accuracy, PubPeer, retraction, and replication, with the reference summary collecting several of them into a single table.