Redis O(1) ID Cache: BaseMap and Cache Services

During a backtest, backtest-kit issues thousands of readXData() calls per second—one for every adapter context touched by the strategy on each simulated tick. If every call hit MongoDB with a compound-field query, even a well-indexed collection would accumulate meaningful latency across millions of ticks. The Redis ID cache solves this by mapping each compound context (exchange + strategy + symbol) to its MongoDB _id string. A Redis GET is O(1) regardless of collection size, and the subsequent MongoDB findById lookup by primary key is O(log n) on the _id index rather than a full compound-index traversal. The net effect is that the vast majority of reads never touch MongoDB’s collection scan path at all.

Why an ID Cache Rather Than a Value Cache

Caching the full document value would create a two-write problem on every update: you would need to atomically update MongoDB and invalidate or replace the Redis value, with no easy way to guarantee the two stay in sync across process restarts or crashes. Caching only the immutable _id avoids this entirely. The _id never changes after insertion, so it can be cached indefinitely with no invalidation logic. On a cache hit, the system fetches the current document from MongoDB by primary key—guaranteeing it always reads the latest value from the source of truth.

BaseMap: Redis Abstraction for All 15 Cache Services

All 15 cache services extend BaseMap, a dependency-injection factory class that wraps ioredis with a Map-like async interface:

const ITERATOR_BATCH_SIZE = 100;
const DEFAULT_TTL_EXPIRE_SECONDS = 5 * 60; // 5 minutes default

export const BaseMap = factory(
  class BaseMap {
    constructor(
      readonly connectionKey: string,
      readonly ttlExpireSeconds: number = DEFAULT_TTL_EXPIRE_SECONDS,
    ) {}

    _getItemKey(key: string): string {
      return `${this.connectionKey}:${key}`;
    }

    async set(key: string, value: unknown): Promise<void> { ... }
    async get(key: string | null): Promise<unknown | null> { ... }
    async delete(key: string): Promise<void> { ... }
    async has(key: string): Promise<boolean> { ... }
    async clear(): Promise<void> { ... }
    async toArray(): Promise<[string, unknown][]> { ... }
    async *iterate(): AsyncIterableIterator<readonly [string, unknown]> { ... }
    async *keys(): AsyncIterableIterator<string> { ... }
    async *values(): AsyncIterableIterator<unknown> { ... }
    async size(): Promise<number> { ... }
  }
);

Key Namespacing

Every Redis key stored by a cache service is prefixed with the service’s connectionKey:

_getItemKey(key: string): string {
  return `${this.connectionKey}:${key}`;
}

This means two services with different connectionKey values can never collide in Redis, even if they store entries with identical sub-keys. For example, signal_cache:binance:macd:BTCUSDT and risk_cache:binance:macd:BTCUSDT are distinct Redis keys despite sharing the same context tuple.

TTL Behavior

BaseMap accepts a ttlExpireSeconds parameter in its constructor. When set to any positive integer, the set method applies a Redis EXPIRE command after writing. All 15 production cache services pass -1, which disables expiry entirely—cached IDs are valid for the lifetime of the process and persist across Redis reconnects:

// TTL = -1 means no expiry; keys live until explicitly deleted or Redis is flushed
export class SignalCacheService extends BaseMap(REDIS_KEY, -1) { ... }

The DEFAULT_TTL_EXPIRE_SECONDS of 5 minutes applies only when BaseMap is used directly without a TTL argument. All 15 named cache services explicitly pass -1 to opt out of expiry.

SCAN-Based Bulk Operations

Methods that operate on multiple keys—clear(), toArray(), iterate(), keys(), values(), size()—use Redis SCAN internally rather than KEYS. This avoids blocking the Redis event loop on large keyspaces. The cursor-based scan is batched at ITERATOR_BATCH_SIZE = 100 keys per round trip, matching the Redis recommendation for incremental iteration.

SignalCacheService: A Detailed Walk-Through

SignalCacheService is the canonical example of the cache pattern used across all 15 services:

const REDIS_KEY = "signal_cache";

export class SignalCacheService extends BaseMap(REDIS_KEY, -1) {
  private _cacheKey(
    symbol: string,
    strategyName: string,
    exchangeName: string,
  ): string {
    return `${exchangeName}:${strategyName}:${symbol}`;
  }

  public async hasSignalId(symbol, strategyName, exchangeName): Promise<boolean> {
    return await this.has(this._cacheKey(symbol, strategyName, exchangeName));
  }

  public async getSignalId(symbol, strategyName, exchangeName): Promise<string | null> {
    const id = await super.get(this._cacheKey(symbol, strategyName, exchangeName));
    return id ?? null;
  }

  public async setSignalId(row: ISignalRowDoc): Promise<string> {
    await super.set(
      this._cacheKey(row.symbol, row.strategyName, row.exchangeName),
      row.id,
    );
    return row.id;
  }
}

The full Redis key for a signal entry is therefore:

signal_cache:binance:macd_cross:BTCUSDT
              │        │           │
              │        │           └── symbol
              │        └────────────── strategyName
              └─────────────────────── exchangeName

And the value stored at that key is the MongoDB _id string (e.g., "6627f3e1a2b3c4d5e6f70001").

The Complete Read Path

SignalDbService.findByContext illustrates the full two-tier lookup:

public findByContext = async (
  symbol: string,
  strategyName: string,
  exchangeName: string,
) => {
  try {
    const cachedId = await this.signalCacheService.getSignalId(
      symbol, strategyName, exchangeName,
    );
    if (cachedId) {
      // O(1) Redis hit → O(log n) MongoDB _id lookup
      return await super.findById(cachedId);
    }
  } catch {
    void 0; // cache miss — fall through to compound query
  }

  // Cache miss: compound-field query
  const result = await super.findByFilter({ symbol, strategyName, exchangeName });
  if (result) {
    await this.signalCacheService.setSignalId(result); // backfill cache
  }
  return result;
};

Redis lookup

Call getSignalId(symbol, strategyName, exchangeName), which issues a single Redis GET on the namespaced key. This is O(1) regardless of how many documents exist in the MongoDB collection.

Cache hit: MongoDB findById

If a cached _id is found, call super.findById(cachedId). MongoDB resolves this with an index seek on the _id B-tree—the fastest possible document lookup path.

Cache miss: compound query

If Redis returns nothing (or throws), fall back to super.findByFilter({ symbol, strategyName, exchangeName }). This traverses the compound index on those three fields.

Cache backfill

After a successful compound query, call setSignalId(result) to populate the cache. All subsequent reads for this context will take the fast path.

The try/catch around the cache lookup is intentional. A transient Redis error (e.g., during reconnect) should degrade gracefully to the MongoDB path rather than surfacing an error to the strategy.

Redis Connection Keepalive and Timeout

The Redis client in redis.ts sends a 30-second keepalive ping to prevent the server from closing idle connections:

// redis.ts (excerpt)
setInterval(async () => {
  await redis.ping();
}, 30_000);

This is particularly important in Docker environments where an intermediate load balancer or firewall may aggressively close TCP connections that have been idle for more than 60 seconds. RedisService.waitForInit enforces a 15-second connection timeout. If the ready event is not received within 15 000 ms, waitForInit.clear() is called to reset the singleshot memoization, allowing the next caller to attempt initialization again:

// RedisService (excerpt)
const CONNECTION_TIMEOUT = 15_000;

const result = await Promise.race([
  waitForConnect(redis, this),
  sleep(CONNECTION_TIMEOUT).then(() => TIMEOUT_SYMBOL),
]);
if (result === TIMEOUT_SYMBOL) {
  this.waitForInit.clear(); // reset so the next call retries
  throw new Error("Redis connection timeout");
}

MongooseService.waitForInit applies the same 15-second timeout, also calling waitForInit.clear() on expiry so that transient connection failures are recoverable.

All 15 Cache Services and Their Redis Keys

Each service owns one connectionKey, which becomes the namespace prefix for all keys it stores.

Cache Service	Redis Key Prefix	Context Sub-key Format
`candleCacheService`	`candle_cache`	`exchangeName:symbol:interval:timestamp`
`signalCacheService`	`signal_cache`	`exchangeName:strategyName:symbol`
`scheduleCacheService`	`schedule_cache`	`exchangeName:strategyName:symbol`
`riskCacheService`	`risk_cache`	`exchangeName:riskName`
`partialCacheService`	`partial_cache`	`exchangeName:strategyName:symbol:signalId`
`breakevenCacheService`	`breakeven_cache`	`exchangeName:strategyName:symbol:signalId`
`storageCacheService`	`storage_cache`	`backtest\|live:signalId`
`notificationCacheService`	`notification_cache`	`backtest\|live:notificationId`
`logCacheService`	`log_cache`	`entryId`
`measureCacheService`	`measure_cache`	`bucket:entryKey`
`intervalCacheService`	`interval_cache`	`bucket:entryKey`
`memoryCacheService`	`memory_cache`	`signalId:bucketName:memoryId`
`recentCacheService`	`recent_cache`	`backtest\|live:exchangeName:strategyName:frameName:symbol`
`stateCacheService`	`state_cache`	`signalId:bucketName`
`sessionCacheService`	`session_cache`	`exchangeName:strategyName:frameName`

To inspect the cache state during development, use the Redis CLI with the SCAN command and a match pattern:

redis-cli SCAN 0 MATCH "signal_cache:*" COUNT 100

This incrementally lists all signal cache keys without blocking Redis.

Get Started

Architecture

Configuration

Guides

Why an ID Cache Rather Than a Value Cache

BaseMap: Redis Abstraction for All 15 Cache Services

Key Namespacing

TTL Behavior

SCAN-Based Bulk Operations

SignalCacheService: A Detailed Walk-Through

The Complete Read Path

Redis Connection Keepalive and Timeout

All 15 Cache Services and Their Redis Keys

Build docs developers (and LLMs) love

Get Started

Architecture

Configuration

Guides

Documentation Index

​Why an ID Cache Rather Than a Value Cache

​BaseMap: Redis Abstraction for All 15 Cache Services

​Key Namespacing

​TTL Behavior

​SCAN-Based Bulk Operations

​SignalCacheService: A Detailed Walk-Through

​The Complete Read Path

​Redis Connection Keepalive and Timeout

​All 15 Cache Services and Their Redis Keys

Build docs developers (and LLMs) love

Why an ID Cache Rather Than a Value Cache

BaseMap: Redis Abstraction for All 15 Cache Services

Key Namespacing

TTL Behavior

SCAN-Based Bulk Operations

SignalCacheService: A Detailed Walk-Through

The Complete Read Path

Redis Connection Keepalive and Timeout

All 15 Cache Services and Their Redis Keys