Detect a file manifest / table-of-contents masquerading as tabular data

A manifest (e.g. a "table of contents" CSV) is structurally a valid tabular file, so extension-based classification treats it as data. It is distinguished from real research data by content, using the repository's own file list as ground truth: a manifest has a column in which most values name other files in the repository. This is name- and header-agnostic – it does not rely on the file or its columns being called anything in particular.

Usage

data_is_manifest(df, repo_files, threshold = 0.8, min_exts = 2L)

Arguments

df

a data.frame (the read tabular file)

repo_files

a character vector of the other file names/paths in the same repository (basenames are compared)

threshold

minimum fraction of a column's non-empty values that must resolve to repository files

min_exts

minimum number of distinct referenced file extensions

Value

TRUE when df looks like a file manifest, else FALSE.

Details

To avoid demoting genuine data that merely references assets (e.g. a stimulus column of image filenames), a candidate column must both reach the threshold of file references and reference at least min_exts distinct file extensions (a manifest points across code/data/docs; an asset column is usually one extension).

Examples

files <- c("data.csv", "analysis.R", "codebook.pdf")
data_is_manifest(data.frame(file = files, size = c(10, 2, 5)), files)
#> [1] TRUE
data_is_manifest(data.frame(id = 1:3, rt = c(512, 640, 498)), files)
#> [1] FALSE