The ref_accuracy module looks for incoherent references. A reference is incoherent when the details the paper cites for it — the title, authors, journal, or year — do not match the record that the reference’s own DOI points to. If you cite a paper as appearing in one journal but its DOI resolves to a different journal, something is wrong: a wrong DOI, a mis-named journal, or (increasingly) an AI-fabricated citation.
The module does not claim a flagged reference is fabricated. It surfaces the discrepancy so you can check the original source and decide what happened.
Only references that supply their own DOI are checked. A reference with no DOI is not checked, because the only record we could find for it comes from a CrossRef title search, which frequently returns the wrong paper — so a “mismatch” would be CrossRef’s error, not the citation’s. References without a DOI are listed separately, with a suggested DOI offered only when CrossRef found a confident match.
This module relies on the paper’s bib_match table, which holds reference metadata retrieved from CrossRef. It is populated when you convert a paper with crossref_lookup = TRUE, or you can add or refresh it at any time with add_bib_match(). Building it makes live network calls, so you need an internet connection.
paper <- add_bib_match(demopaper())
mo <- module_run(paper, "ref_accuracy")
mo$traffic_light#> [1] "green"mo$summary_text#> [1] "We checked 2 references that supplied a DOI and found 0 incoherent references. A further 1 reference were matched to a DOI via CrossRef (0 incoherent, tentative). 2 references had no DOI and could not be checked."The summary_table reports, per paper, how many references were checked (had their own DOI), how many were flagged as incoherent, and how many had no DOI and so could not be checked:
mo$summary_table |>
knitr::kable()| paper_id | refs_provided_doi | refs_crossref_doi | not_checked | incoherent_doi | incoherent_match |
|---|---|---|---|---|---|
| to_err_is_human | 2 | 1 | 2 | 0 | 0 |
The full table has one row per reference, with a tier column ("provided" for references that supplied their own DOI, "none" for references without one) and an incoherent flag, alongside the individual *_mismatch columns:
| bib_id | tier | incoherent | title_mismatch | author_mismatch | container_mismatch | year_mismatch |
|---|---|---|---|---|---|---|
| 1 | provided | FALSE | FALSE | FALSE | FALSE | FALSE |
| 2 | provided | FALSE | FALSE | FALSE | FALSE | FALSE |
| 3 | crossref | FALSE | FALSE | TRUE | FALSE | FALSE |
| 0 | none | FALSE | NA | NA | NA | NA |
| 4 | none | FALSE | NA | NA | NA | NA |
papers <- add_bib_match(psychsci[1:10])
mo <- module_run(papers, "ref_accuracy")
mo$summary_tableFor each checked reference the module compares five fields against the DOI’s record. Two of them are treated as reliable signals and flag a reference on their own; the other two are more often disturbed by imperfect PDF parsing and are treated as weaker signals.
| Field | Signal | Notes |
|---|---|---|
| DOI | reliable | only when the reference printed a DOI that differs from the record |
| Journal | reliable | abbreviation-tolerant (ISO-4, &/and, the) |
| Year | reliable | allows a year_tolerance difference (online-first vs print) |
| Title | weaker | character similarity, ignoring case/punctuation/footnotes |
| Author | weaker | leading surnames only; diacritics ignored |
A reference is flagged when any reliable field mismatches, or when at least min_mismatches of the weaker fields (title, author) mismatch. By default min_mismatches = 2, so a lone title or author difference — which is usually a parsing artefact — is not enough on its own.
ref_accuracy exposes the thresholds behind each check so you can tune the trade-off between catching more errors and raising fewer false alarms.
min_mismatches (default 1)How many of the parsing-sensitive fields (title and author) must disagree before a reference is flagged. The default of 1 flags a single title or author mismatch, which catches roughly twice as many genuine errors. The extra false positives this adds are minor compared with the false positives that already come from imperfect reference parsing, so the more sensitive setting is the better trade-off. Set it to 2 to be more conservative and require two of these fields to disagree before flagging.
# default: a single title or author mismatch is flagged
module_run(paper, "ref_accuracy", min_mismatches = 1)
# conservative: require two parsing-sensitive fields to disagree
module_run(paper, "ref_accuracy", min_mismatches = 2)year_tolerance (default 1)How many years the cited year may differ from the record before the year is flagged. The default allows a one-year difference, because the online-first and print publication years of an article routinely differ by a year. Set it to 0 to require an exact match, or higher to be more lenient.
# require the cited year to match exactly
module_run(paper, "ref_accuracy", year_tolerance = 0)title_similarity (default 0.7)The minimum character-level similarity (0–1, after stripping case, punctuation, footnote markers and diacritics) for a cited title to count as matching the record’s title. Lower values tolerate more formatting noise; a title below the threshold is flagged. Raise it to be stricter about title differences.
# stricter: smaller title differences are flagged
module_run(paper, "ref_accuracy", title_similarity = 0.85)suggest_score (default 70)For references without a DOI, the minimum CrossRef relevance score for a DOI found by title search to be offered as a suggestion in the “references without a DOI” list. Low-scoring matches are usually the wrong paper, so they are withheld. Raise it to suggest fewer (more conservative) DOIs, lower it to suggest more.
# only suggest a DOI when CrossRef is very confident
module_run(paper, "ref_accuracy", suggest_score = 90)The two examples below are genuine citation errors found in the psychsci corpus. Both papers also produce false positives (see the next section), which is the normal situation: a real error usually sits among a few parsing artefacts, and the value of the module is making the small set of candidates visible so you can check them.
In one paper, Webster, Georgeson, and Webster (2002), “Neural adjustments of image blur”, is cited as appearing in Nature Neuroscience — which is correct. But the DOI attached to the reference, 10.1167/1.3.441, resolves to a Journal of Vision article. The cited journal is right; the DOI is wrong (the correct DOI is 10.1038/nn906). The module catches the inconsistency through the journal mismatch:
| container.orig | container.match | doi.match |
|---|---|---|
| Nature Neuroscience | Journal of Vision | 10.1167/1.3.441 |
A reader who followed that DOI would land on the wrong paper — exactly the kind of error the check is for.
In another paper, Võ, Boettcher, and Draschkow (2019) is cited as appearing in a journal called “Attention & Perception”. The DOI resolves to the correct paper, with the correct title and year, but the real journal is Current Opinion in Psychology — “Attention & Perception” is not a real journal. Here the cited details are wrong and the DOI is right:
| container.orig | container.match | year.orig | year.match |
|---|---|---|---|
| Attention & Perception | Current Opinion in Psychology | 2019 | 2019 |
A flagged reference is a candidate to check, not a confirmed error. From reviewing many flags across fields, the discrepancies fall into a handful of recurring causes:
Because most flags are data or parsing issues rather than authorial mistakes, treat each one as “look at the original source”, not as a confirmed error. A journal mismatch on an older reference is often a renaming; a journal and year mismatch together more often points to a wrong DOI.
Many of the false positives above — merged references, journal names pulled from the wrong place, DOIs attached to the wrong entry — come from how references are currently extracted from the PDF, not from the citations themselves. We are working on a more accurate reference-import tool, which should remove most of these artefacts and make the flags that remain far more likely to be genuine errors.
ref_accuracy takes the paper argument plus the five tuning parameters described above: min_mismatches, year_tolerance, title_similarity, max_authors, and suggest_score.
This is one of the reference-section checks. The reference summary module collects the results of several of them — accuracy, PubPeer, retraction, and replication — into a single per-reference table.