Skip to content

Fixes and updates for file uploads#42

Merged
suu-yi merged 18 commits intomasterfrom
hdf5-attr-patch
Mar 9, 2026
Merged

Fixes and updates for file uploads#42
suu-yi merged 18 commits intomasterfrom
hdf5-attr-patch

Conversation

@suu-yi
Copy link
Collaborator

@suu-yi suu-yi commented Mar 6, 2026

Upload system overhaul

  • Moved HTTP primitives (.make_req, .url_path, constants) from client.R to api.R
  • Added direct R2/S3 presigned URL upload path alongside server-proxy uploads
  • Initiate request now sends file_size_bytes; completion sends parts array (ETag + PartNumber) for R2 uploads
  • Presigned URL count validated against expected chunk count
  • Added .put_to_presigned_url with retry, ETag validation, and proper error classification
  • Raised obs_duckdb upload limit from 100MB to 2GB; max_workers default from 4 to 6

Progress bars

  • Upload progress: cli bar for sequential, progressr + handler_cli for parallel (graceful fallback to sequential when progressr not installed)
  • Artifact writing: consolidated cli progress bar for vars.h5 (tracks both normalized + raw batches); cli::cli_progress_step spinner for obs.duckdb
  • progressr and future added to Suggests

Preparation & metadata

  • Replaced all print() calls in PrepareCyteTypeR with log_info()
  • Added max_metadata_categories parameter to skip high-cardinality columns during metadata aggregation
  • pcent_batch_size default raised from 2000 to 5000
  • HDF5 attributes fix

@suu-yi suu-yi changed the title Fix vars.h5 attribute to scalar for python Fixes and updates for file uploads Mar 9, 2026
@suu-yi suu-yi merged commit 00819d0 into master Mar 9, 2026
4 checks passed
@suu-yi suu-yi deleted the hdf5-attr-patch branch March 10, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant