Skip to contents

Is a read-in data frame a usable rectangular dataset?

Source: R/data_check_helpers.R
dot-tabular_usable.Rd

readxl/read.delim will happily read a human-formatted coding worksheet (interleaved "Code N: description" free-text columns + sparse 0/1 indicators, summary/legend rows at the bottom) into a data frame – but the result is not a rectangular dataset: most "columns" are prose annotations and most cells are structurally empty. Extracting columns from such a file yields junk (the "Code"-prefixed columns) and sending them to the LLM wastes calls on non-data.

Usage

.tabular_usable(facets, df)

Arguments

facets

a list of per-column facet lists, as produced by data_col_facets() (one per column, in column order).

df

the read-in data frame the facets describe.

Value

a list with usable (logical) and, when FALSE, a human-readable reason naming the signals that fired.

Details

This detects that case from facets data_check has already computed, using two signals combined with tiered rules (not a single AND-gate, so an extreme value on one axis can exclude a file on its own):

  • prose fraction – share of columns that are free text: representation is "text", the concept is not a recognised text kind (id/date/timestamp), and the column is high-cardinality (distinct/non-missing > 0.5), i.e. a genuine free-text field, not a small set of category labels.

  • missingness fraction – share of columns that are >50% missing.

Exclude when the file is almost entirely empty, or overwhelmingly free text, or moderately both. A file with an ordinary structure (a few open-response columns, or a legitimately sparse but numeric design) trips none of these.

Examples

df <- read.csv(file.path(demofile("repo"), "data", "stroop_data.csv"))
facets <- Map(data_col_facets, names(df), df)
.tabular_usable(facets, df)
#> $usable
#> [1] TRUE
#> 
#> $reason
#> [1] NA
#>