Skip to main content
The ByteStream service provides streaming APIs for transferring large blobs, complementing the batch operations in the CAS service.

​
Overview

ByteStream enables:
  • Streaming uploads of large files
  • Resumable downloads
  • Efficient data transfer for blobs too large for batch operations
Key features:
  • Chunked streaming (no size limits)
  • Resumable uploads
  • Read offsets and limits

​
Configuration

​
instance_name
string
required
The instance name identifying this ByteStream endpoint
​
cas_stores
object
required
Map of instance names to CAS store references
{
  services: {
    bytestream: {
      cas_stores: {
        "main": "MAIN_CAS_STORE"
      }
    }
  }
}

​
gRPC Methods

​
Read

Download a blob in chunks. Request:
​
resource_name
string
required
Format: {instance_name}/blobs/{hash}/{size}Example: main/blobs/abc123.../1048576
​
read_offset
int64
default:0
Byte offset to start reading from
​
read_limit
int64
default:0
Maximum bytes to read (0 = all remaining)
Response (stream):
​
data
bytes
Chunk of blob data
Example with grpcurl: 
grpcurl -plaintext \
  -d '{
    "resource_name": "main/blobs/abc123.../1048576",
    "read_offset": 0,
    "read_limit": 0
  }' \
  localhost:50051 google.bytestream.ByteStream/Read
Use read_offset to implement resumable downloads after connection failures

​
Write

Upload a blob in chunks. Request (stream):
​
resource_name
string
required
Format: {instance_name}/uploads/{uuid}/blobs/{hash}/{size}The UUID must be client-generated and unique per upload.
​
write_offset
int64
required
Byte offset of this chunk (must be sequential)
​
finish_write
boolean
required
True on the final chunk
​
data
bytes
required
Chunk data
Response:
​
committed_size
int64
Total bytes successfully written
Upload flow:
1

Generate UUID

Create a unique UUID for this upload session
2

Send chunks

Stream chunks with sequential write_offset values
3

Finish upload

Set finish_write=true on the last chunk
Example (first chunk): 
{
  "resource_name": "main/uploads/550e8400-e29b-41d4-a716-446655440000/blobs/abc123.../1048576",
  "write_offset": 0,
  "finish_write": false,
  "data": "<base64 encoded chunk>"
}
Example (last chunk): 
{
  "resource_name": "main/uploads/550e8400-e29b-41d4-a716-446655440000/blobs/abc123.../1048576",
  "write_offset": 1048000,
  "finish_write": true,
  "data": "<base64 encoded chunk>"
}
If the final digest doesn’t match the hash in the resource name, the upload fails

​
QueryWriteStatus

Check the status of an incomplete upload. Request:
​
resource_name
string
required
The upload resource name
Response:
​
committed_size
int64
Bytes successfully written so far
​
complete
boolean
True if upload is finished
Use QueryWriteStatus to resume interrupted uploads from the last committed offset

​
Resource Name Format

ByteStream uses specific resource name formats: Read: 
{instance_name}/blobs/{hash}/{size}
Write: 
{instance_name}/uploads/{uuid}/blobs/{hash}/{size}
Compressed read: 
{instance_name}/compressed-blobs/{compressor}/{uncompressed_hash}/{uncompressed_size}

​
Compression Support

ByteStream supports compressed transfers: Compressor values:
  • identity - No compression
  • zstd - Zstandard compression
  • deflate - DEFLATE compression
Example compressed read: 
main/compressed-blobs/zstd/abc123.../1048576

​
Chunk Size Recommendations

Recommended chunk sizes:
  • Small files (<1MB): Use BatchUpdateBlobs instead
  • Medium files (1-100MB): 256KB - 1MB chunks
  • Large files (>100MB): 1-4MB chunks

​
Error Codes

CodeDescription
NOT_FOUNDBlob or upload session not found
INVALID_ARGUMENTInvalid resource name or offset
OUT_OF_RANGERead offset beyond blob size
FAILED_PRECONDITIONNon-sequential write offset
RESOURCE_EXHAUSTEDDisk full
DATA_LOSSHash mismatch on completed upload

​
Implementation Details

From nativelink-service/src/bytestream_server.rs, the ByteStream server wraps the underlying CAS stores and handles chunked streaming.
The ByteStream service follows the Google ByteStream API specification

Build docs developers (and LLMs) love

Get started for freeTalk to us