Documentation Index
Fetch the complete documentation index at: https://mintlify.com/sachnun/hugbucket/llms.txt
Use this file to discover all available pages before exploring further.
What is HugBucket?
HugBucket is a self-hosted gateway that translates standard S3 and FTP protocols into the Hugging Face Storage Bucket API. It lets any tool that speaks S3 (AWS CLI, boto3, S3 Browser, rclone) or FTP read and write directly to your Hugging Face repositories — no code changes required. Under the hood HugBucket implements the full Xet content-addressable storage protocol in pure Python: content-defined chunking, BLAKE3 hashing, LZ4 compression, and parallel xorb uploads. This is the same storage engine Hugging Face uses natively, so data written through HugBucket is fully compatible with thehuggingface_hub Python library and the HF web UI.
Quick Start
Get HugBucket running in two
docker run commandsS3 Protocol
Full S3 REST API — list, get, put, delete, copy, multipart upload
FTP Protocol
FTP access with path mapping as
/<bucket>/<key>Architecture
How HugBucket maps S3/FTP operations onto Xet storage
Key features
S3-compatible API
Works with AWS CLI, boto3, rclone, S3 Browser, and any other S3-compatible client out of the box.
FTP gateway
Classic FTP access with
/<bucket>/<key> path mapping — useful for legacy tools and simple file transfers.Native Xet storage
Pure Python implementation of the Xet CAS protocol: CDC chunking, BLAKE3 hashing, LZ4 compression.
Docker-first
A single container image runs either gateway. Switch between S3 and FTP with the
MODE environment variable.Streaming downloads
Range-aware streaming with byte-range request support. Only fetches the xorb chunks that cover the requested range.
Multipart uploads
Full S3 multipart upload protocol with idempotent completion and automatic stale-upload cleanup.
Server-side copy
CopyObject reuses the existing content hash — no re-download or re-upload of bytes.Built-in caching
LRU caches for xorb chunks (512 MiB), file metadata, and reconstruction plans to minimize round-trips.
Supported S3 operations
HugBucket implements the following S3 REST API operations:| Operation | Method | Description |
|---|---|---|
ListBuckets | GET / | List all HF Storage Buckets |
CreateBucket | PUT /{bucket} | Create a new bucket |
DeleteBucket | DELETE /{bucket} | Delete a bucket |
HeadBucket | HEAD /{bucket} | Check bucket existence |
GetBucketLocation | GET /{bucket}?location | Return bucket region |
ListObjectsV2 | GET /{bucket}?list-type=2 | List objects with prefix/delimiter |
PutObject | PUT /{bucket}/{key} | Upload an object |
GetObject | GET /{bucket}/{key} | Download an object (supports Range) |
HeadObject | HEAD /{bucket}/{key} | Get object metadata |
DeleteObject | DELETE /{bucket}/{key} | Delete an object |
DeleteObjects | POST /{bucket}?delete | Batch delete objects |
CopyObject | PUT + x-amz-copy-source | Server-side copy |
| Multipart upload | POST, PUT, DELETE | Initiate, upload parts, complete, abort |
Requirements
- A Hugging Face account with an access token (
HF_TOKEN) - Docker (recommended) or Python 3.14+ with uv