Download the files listed by repo_check() — download_repo

Fetches the bytes for a table of repository files (as produced by repo_check()), writing each into a per-session temp directory or a persistent on-disk cache, and fills in file_location for every file it successfully retrieves. For Dryad (stash:download endpoint), GitHub (zipball) and GitLab (archive.zip), a whole-repo archive download is used instead of one HTTP request per file. None of these archives reports its size in advance, so the size is estimated from the listed file sizes: an archive with no listed sizes, or an estimate above twice max_download_size, is not used, and a download that grows past twice max_download_size is stopped. A repo whose archive is not used or fails falls back to file-by-file fetching automatically, as do all other hosts.

Usage

download_repo_files(
  files,
  max_file_size = 100,
  max_download_size = 500,
  max_files_per_repo = Inf,
  repo_file_counts = NULL,
  zip_timeout_s = 120,
  cache = FALSE,
  skip_on_api_limit = FALSE,
  pb = NULL
)

Arguments

files: a data.frame of files (as returned by repo_check()'s table): needs repo_url, file_url, and file_path or file_name
max_file_size: largest single file to download, in MB
max_download_size: largest total cached footprint per repository, in MB
max_files_per_repo: largest file COUNT a single repository may have before it is refused outright (same "gated" reporting as the size caps above), default Inf (no cap). Unlike max_file_size/max_download_size, this catches a repo whose FILE COUNT (not byte size) makes it impractical to process one file at a time – confirmed live 2026-09-07 against a real Zenodo record (SAPFLUXNET, ~4000 files) whose per-file classification and full-content read (in data_check, downstream of this function) took hours and multiple GB of memory for a single paper's repository.
repo_file_counts: optional named vector/table, repo_url -> TRUE total row count for that repo in the caller's FULL listing, before any filtering to "still need downloading". files here is routinely a filtered subset (data_check()/code_check() only pass rows that are wanted and not yet on disk), so length(idx) for a repo within files undercounts its real size – confirmed live 2026-09-09: the SAME SAPFLUXNET record above (~4000 files total) had under max_files_per_repo "still need downloading" rows on one run, so the gate never fired, and several hundred individual files were then requested from a host that was, at the time, answering every one with a persistent (not transient) 403 – each retried independently, compounding into a multi-hour stall with no single wait long enough to look like the real problem. When a repo's URL is not a name in this vector (e.g. NULL, the default, for a caller that has not been updated to pass it), falls back to length(idx) exactly as before.
zip_timeout_s: timeout (seconds) for a whole-repo zip download attempt before falling back to file-by-file fetching
cache: if TRUE, write into the lasting cache (see repo_cache_dir(); survives across sessions, never cleared automatically — see repo_cache_clear()); if FALSE (default), write into a per-session temp directory that R removes on exit
skip_on_api_limit: if FALSE (the default), a download that hits a host-confirmed exhausted rate limit (a 429 whose response carries RateLimit-Remaining: 0/X-RateLimit-Remaining: 0 and a parseable reset time — currently known to apply to Dryad, Zenodo, GitHub, and GitLab; other hosts do not expose this and are unaffected either way) waits for the host's own stated reset before retrying, however long that is — appropriate for an unattended corpus build, where completeness matters more than wall-clock time. Set TRUE to give up on that file immediately instead: it is recorded in the "failed" attribute with an "API rate limit exhausted: "-prefixed message, filterable from every other kind of failure, so a caller can re-run later (once the quota has reset) targeting only those specific files rather than waiting inline. An ordinary transient refusal (a 429 without a confirmed exhausted-quota signal, a 403/5xx) still retries with ordinary backoff regardless of this setting — it only changes what happens once a real quota exhaustion is confirmed.
pb: an optional progress bar object (see pb()), ticked as files download

Value

files with file_location filled in for every file retrieved (unchanged, i.e. NA, for files that were skipped or failed), and three attributes: "gated" (data.frame: repo_url, message — repositories refused outright by the size caps), "oversize_skipped" (data.frame: repo_url, file_name, file_size — individual files skipped under max_file_size), and "failed" (data.frame: repo_url, file_name, file_url, paper_id, error — files whose download was attempted but errored, e.g. a transient network failure, or — when skip_on_api_limit = TRUE and the host confirmed it — an exhausted API rate limit; see skip_on_api_limit. file_url is carried along so a caller can point someone at the file directly instead of re-deriving the link. paper_id is files$paper_id[i] when files carries that column (e.g. a paperlist batch), NA otherwise — lets a caller attribute a failure back to the paper it belongs to without re-joining on repo_url).

Details

Two independent size gates apply: max_file_size skips oversize files individually (the rest of the repository still downloads); max_download_size is a budget on the repository's TOTAL cached footprint (already-cached plus newly-downloaded), not on a single run's increment — already-cached files are kept for free, then the smallest still-missing files are added until the next one would exceed the budget, so re-running never grows a repository's cache past the budget and the canonical "files we ever download" set is fixed.

Examples

# \donttest{
if (online("raw.githubusercontent.com")) {
  files <- data.frame(
    repo_url = "https://github.com/scienceverse/metacheck",
    file_url = "https://raw.githubusercontent.com/scienceverse/metacheck/main/DESCRIPTION",
    file_name = "DESCRIPTION", file_path = "DESCRIPTION", file_size = 3000)
  # kept in this session's temporary folder (cache = FALSE)
  dl <- download_repo_files(files)
  dl$file_location
}
#> Downloading https://github.com/scienceverse/metacheck as zip (1 file)...
#> [1] "/tmp/RtmpNCKKS0/metacheck-repo-files/github.com_scienceverse_metacheck/DESCRIPTION"
# }