Check downloaded Zenodo files against the file system

Confirms that every file the download planned to save is present, is the size Zenodo reported, and matches the checksum Zenodo published for it.

Usage

.zenodo_verify_downloads(files, download_to)

Arguments

files

the file table being built, with path, size, checksum, and downloaded columns, and optionally extracted

download_to

the folder the record was saved in

Value

files with downloaded corrected against the file system, plus size_on_disk and checksum_ok columns

Details

Up to this point downloaded records only that the transfer step ran. A request that failed, a copy that did not happen, or a truncated write all leave a row marked TRUE with nothing usable on disk. That matters most when archiving a whole record unattended, where nobody is watching each file.

Zenodo publishes an MD5 checksum per file, which catches corruption that a size comparison cannot – a file can be the right length and still be wrong. Hashing uses tools::md5sum() from base R, as import-grobid.R and llm-cache.R already do, so this adds no dependency. Only files Zenodo gave an md5: value for are hashed; for anything else the size check stands on its own.

A row whose extracted count is set is exempt from both checks. Zenodo's size and MD5 describe the zip as published, and that zip was deliberately never downloaded – only chosen members were. Measured against those values every such row would look truncated. The members carry their own integrity check instead: each is verified against the CRC32 stored in the archive's central directory as it is extracted, in .zip_member_fetch().