Redis Store

Overview

The Redis store provides distributed in-memory storage using Redis or Redis-compatible services. It’s ideal for sharing cache across multiple NativeLink instances while maintaining high performance. The store supports Redis standalone, cluster, and sentinel modes.

Pairs well with SizePartitioning and FastSlow stores. Ideal for small objects as most Redis services have a max upload size of 256MB-512MB.

Use Cases

Distributed action cache: Share build action results across multiple workers
Small object caching: Cache frequently accessed small objects (< 256MB)
Shared state: Coordination between NativeLink instances
Fast distributed tier: Alternative to memory store that survives restarts
Scheduler backend: Store scheduler state in Redis with pub/sub support

Performance Characteristics

Read performance: Sub-millisecond for network-local Redis (1-5ms typical)
Write performance: Sub-millisecond for small objects
Durability: Configurable (in-memory, RDB snapshots, AOF)
Availability: High with Redis Sentinel or Cluster
Size limits: Most Redis services limit objects to 256MB-512MB

Configuration

addresses

string[]

required

List of Redis server addresses in the format:redis://[username:password@]host:port[/database_id]Examples:

redis://127.0.0.1:6379/ (local, default DB)
redis://username:password@redis-server:6380/99 (authenticated, DB 99)
redis://redis-cluster:6379/ (cluster node)

For cluster mode, provide multiple cluster node addresses.

mode

RedisMode

default:"standard"

Redis operation mode.Available options:

"standard": Standard Redis (not cluster or sentinel)
"cluster": Redis Cluster mode
"sentinel": Redis Sentinel mode

Default: "standard"

key_prefix

string

default:""

Optional prefix to prepend to all keys in this store.Useful for:

Organizing data by namespace
Multi-tenant configurations
Separating environments (dev, staging, prod)

Example: "nativelink:cas:"Default: "" (no prefix)

read_chunk_size

number

default:"65536"

The amount of data to read from Redis at a time (in bytes).Used to limit memory usage when reading large objects and prevent command_timeout_ms from being triggered on slow connections.Important: If this value is too high, the command_timeout_ms might be triggered if latency or throughput to Redis is too low.Default: 64KB (65536 bytes)

connection_pool_size

number

default:"3"

The number of connections to keep open to the Redis server(s).More connections can improve throughput but increase resource usage.Default: 3

max_chunk_uploads_per_update

number

default:"10"

The maximum number of upload chunks to allow per update.Used to limit memory usage when uploading large objects. A good rule of thumb:AVAIL_MEMORY / (read_chunk_size * max_chunk_uploads_per_update) ≈ THEORETICAL_MAX_CONCURRENT_UPLOADSDivide AVAIL_MEMORY by ~10 to account for other memory usage.Default: 10

command_timeout_ms

number

default:"10000"

Time in milliseconds until the Redis store considers a command timed out.This triggers a retry and potentially a reconnection to the Redis server.Default: 10000ms (10 seconds)

connection_timeout_ms

number

default:"3000"

Time in milliseconds until the Redis store considers the connection unresponsive.This triggers a reconnection to the Redis server.Default: 3000ms (3 seconds)

max_client_permits

number

default:"500"

Maximum number of permitted actions to the Redis store at any one time.Prevents timeout issues from too many inflight actions.Default: 500

scan_count

number

default:"10000"

The COUNT value passed when scanning keys in Redis.Used to hint the amount of work per response when scanning large keyspaces.Default: 10000

max_count_per_cursor

number

default:"1500"

Maximum number of items returned per cursor for search indexes.May reduce thundering herd issues with worker provisioner at higher node counts.Default: 1500

retry

Retry

Retry configuration for failed Redis operations.Default configuration:

{
  "max_retries": 0,
  "delay": 0.1,
  "jitter": 0.5
}

See Retry Configuration for details.

experimental_pub_sub_channel

string

Optional Redis channel to publish write events to.If set, every write operation publishes an event to this channel. If unset, writes still occur but events aren’t published.Experimental feature.Default: Not set (no events published)

Deprecated Configuration

The following fields are deprecated. Use the new timeout fields instead.

response_timeout_s

number

deprecated

Deprecated: Use command_timeout_ms instead.The response timeout for Redis connections in seconds.

connection_timeout_s

number

deprecated

Deprecated: Use connection_timeout_ms instead.The connection timeout for Redis connections in seconds.

broadcast_channel_capacity

number

deprecated

Deprecated: No longer used by redis-rs.

Retry Configuration

retry.max_retries

number

default:"0"

Maximum number of retries. Setting to 0 means unlimited retries.Default: 0 (unlimited)

retry.delay

number

default:"0.1"

Base delay in seconds for exponential backoff.Default: 0.1 (100ms)

retry.jitter

number

default:"0.5"

Jitter as a percentage (0.0 to 1.0) to add to delays.Default: 0.5 (50%)

Redis Mode Configuration

Standard Mode

Single Redis instance or simple replication.

{
  "redis_store": {
    "addresses": ["redis://127.0.0.1:6379/"],
    "mode": "standard",
    "max_client_permits": 1000
  }
}

Cluster Mode

Redis Cluster for horizontal scaling and automatic sharding.

{
  "redis_store": {
    "addresses": [
      "redis://node1:6379/",
      "redis://node2:6379/",
      "redis://node3:6379/"
    ],
    "mode": "cluster",
    "max_client_permits": 1000
  }
}

Sentinel Mode

Redis Sentinel for high availability and automatic failover.

{
  "redis_store": {
    "addresses": [
      "redis://sentinel1:26379/",
      "redis://sentinel2:26379/",
      "redis://sentinel3:26379/"
    ],
    "mode": "sentinel",
    "max_client_permits": 1000
  }
}

Configuration Examples

Basic Redis Store

{
  "redis_store": {
    "addresses": ["redis://127.0.0.1:6379/"],
    "max_client_permits": 1000
  }
}

With Authentication and Custom Database

{
  "redis_store": {
    "addresses": ["redis://username:password@redis-server:6380/99"],
    "key_prefix": "nativelink:prod:",
    "max_client_permits": 1000
  }
}

Optimized for Large Objects

{
  "redis_store": {
    "addresses": ["redis://redis-server:6379/"],
    "read_chunk_size": 131072,
    "max_chunk_uploads_per_update": 20,
    "command_timeout_ms": 30000,
    "max_client_permits": 1000
  }
}

Size-Partitioned with Redis for Small Objects

{
  "size_partitioning": {
    "size": "128mib",
    "lower_store": {
      "redis_store": {
        "addresses": ["redis://redis-server:6379/"],
        "key_prefix": "small:",
        "max_client_permits": 1000
      }
    },
    "upper_store": {
      "experimental_cloud_object_store": {
        "provider": "aws",
        "region": "us-west-2",
        "bucket": "large-objects"
      }
    }
  }
}

Redis as Fast Tier with S3 Slow Tier

{
  "fast_slow": {
    "fast": {
      "redis_store": {
        "addresses": ["redis://redis-cluster:6379/"],
        "mode": "cluster",
        "key_prefix": "cache:",
        "max_client_permits": 1000
      }
    },
    "slow": {
      "experimental_cloud_object_store": {
        "provider": "aws",
        "region": "us-east-1",
        "bucket": "build-cache"
      }
    }
  }
}

With Pub/Sub for Distributed Notifications

{
  "redis_store": {
    "addresses": ["redis://redis-server:6379/"],
    "experimental_pub_sub_channel": "nativelink:updates",
    "max_client_permits": 1000
  }
}

Production Configuration with Tuning

{
  "redis_store": {
    "addresses": [
      "redis://redis-cluster-node1:6379/",
      "redis://redis-cluster-node2:6379/",
      "redis://redis-cluster-node3:6379/"
    ],
    "mode": "cluster",
    "key_prefix": "nativelink:prod:ac:",
    "read_chunk_size": 65536,
    "connection_pool_size": 5,
    "max_chunk_uploads_per_update": 10,
    "command_timeout_ms": 10000,
    "connection_timeout_ms": 3000,
    "max_client_permits": 1000,
    "scan_count": 10000,
    "retry": {
      "max_retries": 0,
      "delay": 0.1,
      "jitter": 0.5
    }
  }
}

Size Limitations

Redis object size limits: Most Redis services limit individual values to 256MB-512MB. For larger objects, use Redis with SizePartitioning or as a fast tier with another store as the slow tier.

Recommended Size Limits

Redis Deployment	Max Object Size	Recommended Use
Managed Redis (AWS ElastiCache, etc.)	256MB	Action Cache, small CAS objects
Self-hosted Redis	512MB	Action Cache, medium CAS objects
Redis with size partitioning	No limit	Combined with S3/filesystem for large objects

Best Practices

Use for Action Cache: Redis excels at storing small action cache entries that are frequently accessed.

Combine with size partitioning: Use Redis for objects under 128MB and S3/filesystem for larger objects.

Use Redis Cluster for scale: For high-throughput workloads, Redis Cluster provides horizontal scaling.

Monitor memory usage: Redis stores all data in memory. Monitor usage and configure appropriate eviction policies on the Redis server.

Tune connection pool: Increase connection_pool_size for high-concurrency workloads, but balance against Redis connection limits.

Set key_prefix for multi-tenancy: Use different prefixes for different environments or tenants sharing the same Redis instance.

Configure Redis persistence: Enable RDB snapshots or AOF on Redis server for data durability across restarts.

Performance Tuning

Network Latency

Co-locate with compute: Place Redis close to NativeLink instances to minimize latency
Use connection pooling: Increase connection_pool_size for better throughput
Tune timeouts: Adjust command_timeout_ms based on network characteristics

Memory Management

Set Redis maxmemory: Configure Redis maxmemory and eviction policy
Use LRU eviction: Configure Redis with maxmemory-policy allkeys-lru
Monitor memory: Track Redis memory usage and adjust limits

Throughput Optimization

Increase chunk size: Larger read_chunk_size reduces round trips but increases memory
Tune concurrency: Adjust max_client_permits and max_chunk_uploads_per_update
Use pipelining: Redis store automatically pipelines operations

Common Patterns

Development Environment

{"redis_store": {"addresses": ["redis://localhost:6379/"], "max_client_permits": 100}}

Production Action Cache

{"redis_store": {"addresses": ["redis://redis-cluster:6379/"], "mode": "cluster", "key_prefix": "ac:", "max_client_permits": 2000}}

Hybrid Storage

{"fast_slow": {"fast": {"redis_store": {"addresses": ["redis://redis:6379/"], "max_client_permits": 1000}}, "slow": {"filesystem": {"content_path": "/var/cache/nativelink", "temp_path": "/tmp/nativelink", "eviction_policy": {"max_bytes": "100gb"}}}}}

Services

Store Types

Overview

Use Cases

Performance Characteristics

Configuration

Deprecated Configuration

Retry Configuration

Redis Mode Configuration

Standard Mode

Cluster Mode

Sentinel Mode

Configuration Examples

Basic Redis Store

With Authentication and Custom Database

Optimized for Large Objects

Size-Partitioned with Redis for Small Objects

Redis as Fast Tier with S3 Slow Tier

With Pub/Sub for Distributed Notifications

Production Configuration with Tuning

Size Limitations

Recommended Size Limits

Best Practices

Performance Tuning

Network Latency

Memory Management

Throughput Optimization

Common Patterns

Development Environment

Production Action Cache

Hybrid Storage

Build docs developers (and LLMs) love

Services

Store Types

​Overview

​Use Cases

​Performance Characteristics

​Configuration

​Deprecated Configuration

​Retry Configuration

​Redis Mode Configuration

​Standard Mode

​Cluster Mode

​Sentinel Mode

​Configuration Examples

​Basic Redis Store

​With Authentication and Custom Database

​Optimized for Large Objects

​Size-Partitioned with Redis for Small Objects

​Redis as Fast Tier with S3 Slow Tier

​With Pub/Sub for Distributed Notifications

​Production Configuration with Tuning

​Size Limitations

​Recommended Size Limits

​Best Practices

​Performance Tuning

​Network Latency

​Memory Management

​Throughput Optimization

​Common Patterns

​Development Environment

​Production Action Cache

​Hybrid Storage

Build docs developers (and LLMs) love

Overview

Use Cases

Performance Characteristics

Configuration

Deprecated Configuration

Retry Configuration

Redis Mode Configuration

Standard Mode

Cluster Mode

Sentinel Mode

Configuration Examples

Basic Redis Store

With Authentication and Custom Database

Optimized for Large Objects

Size-Partitioned with Redis for Small Objects

Redis as Fast Tier with S3 Slow Tier

With Pub/Sub for Distributed Notifications

Production Configuration with Tuning

Size Limitations

Recommended Size Limits

Best Practices

Performance Tuning

Network Latency

Memory Management

Throughput Optimization

Common Patterns

Development Environment

Production Action Cache

Hybrid Storage