Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/backtest-kit/backtest-kit-minio-s3-docker/llms.txt

Use this file to discover all available pages before exploring further.

S3 lists objects only in lexicographic order. For most entities that is perfectly fine — keys are deterministic, and point reads by key are all that is needed. But three entities in backtest-kit need newest-first listings of an unbounded number of records: Log, Notification, and Storage. Walking the entire S3 prefix in order and reversing it would be O(bucket size), which grows without bound during a long-running strategy. Redis solves this with a compact time-ordered index that makes newest-first lookups O(limit) regardless of how many objects the bucket holds.

​
Which Entities Use Redis Indexing

Exactly three entities maintain a Redis index alongside their S3 objects:
EntityRedis key prefixConnection service
Loglog-items-connectionLogConnectionService
Notificationnotification-items-connectionNotificationConnectionService
Storagestorage-items-connectionStorageConnectionService
These three share an identical implementation pattern. Each has a *ConnectionService that extends BaseMap and exposes two public methods: register(objectName) and listNewest(limit, prefix). The remaining 13 entities do not need newest-first listings, so they have no Redis index at all.

​
The Minute-Bucket Design

Rather than storing one global sorted set (which would require a ZADD score per entry and a ZREVRANGEBYSCORE query), the index uses one Redis SET per calendar minute. All object names written within the same wall-clock minute land in the same SET. The Redis key for a minute bucket is: 
<entity>-connection:<minute-timestamp-zero-padded>
where <minute-timestamp> is the Unix millisecond timestamp of the minute boundary (produced by alignToInterval(new Date(), "1m")), zero-padded to Number.MAX_SAFE_INTEGER.toString().length digits so that lexicographic and numeric order agree. A separate floor marker key records the very first minute ever written: 
<entity>-connection:floor  →  "<first-minute-timestamp>"
The floor key bounds the backwards walk in listNewest — there is no need to probe minutes before it. The register method for all three connection services uses a single pipelined round trip: 
// src/lib/services/connection/LogConnectionService.ts
public register = async (objectName: string): Promise<void> => {
  this.loggerService.log("logConnectionService register", { objectName });
  const redis = await getRedis();
  // One Redis SET per minute: SADD deduplicates repeated names, the floor
  // marker (first-ever minute) bounds the backwards walk in listNewest
  const minute = alignToInterval(new Date(), "1m").getTime();
  await redis
    .pipeline()
    .sadd(GET_MINUTE_KEY_FN(this.connectionKey, minute), objectName)
    .setnx(GET_FLOOR_KEY_FN(this.connectionKey), String(minute))
    .exec();
};
SADD naturally deduplicates: if register is called twice for the same object name within the same minute (e.g. during a fast backtest replay), the second call is a no-op. SETNX sets the floor key only if it does not already exist, so the floor always reflects the very first minute seen.
The floor marker key is what makes the backwards walk safe and bounded. Without it, listNewest would have to walk all the way back to Unix epoch zero. With it, the walk stops as soon as minute < floor, which is typically just a few hundred minutes into the past even for multi-week backtests.

​
The listNewest Algorithm

listNewest(limit, prefix) walks backwards from the current minute toward the floor, collecting object names until limit is reached. It never issues a SCAN over the full Redis keyspace — all lookups are direct key accesses. 
// src/lib/services/connection/LogConnectionService.ts
public listNewest = async (limit: number, prefix = ""): Promise<string[]> => {
  this.loggerService.log("logConnectionService listNewest", { limit, prefix });
  const redis = await getRedis();
  const floorRaw = await redis.get(GET_FLOOR_KEY_FN(this.connectionKey));
  if (!floorRaw) {
    return [];
  }
  const floor = Number(floorRaw);
  // We know the current time — walk backwards minute by minute with direct
  // key lookups (no SCAN over the keyspace), pipelined per WALK_BATCH_SIZE
  let minute = alignToInterval(new Date(), "1m").getTime();
  const seen = new Set<string>();
  const names: string[] = [];

  const collect = (members: string[]): boolean => {
    for (const name of members) {
      if (prefix && !name.startsWith(prefix)) {
        continue;
      }
      if (seen.has(name)) {
        continue;
      }
      seen.add(name);
      names.push(name);
      if (names.length >= limit) {
        return true;
      }
    }
    return false;
  };

  while (minute >= floor && names.length < limit) {
    const batch: number[] = [];
    while (batch.length < WALK_BATCH_SIZE && minute >= floor) {
      batch.push(minute);
      minute -= MS_PER_MINUTE;
    }
    // Cheap cardinality probe: empty minutes are skipped without
    // transferring a single member
    const cardPipeline = redis.pipeline();
    for (const ts of batch) {
      cardPipeline.scard(GET_MINUTE_KEY_FN(this.connectionKey, ts));
    }
    const cards = await cardPipeline.exec();
    if (!cards) {
      break;
    }
    // Pipeline results follow command order: minutes descend, newest first
    for (let i = 0; i < batch.length; i++) {
      const [error, card] = cards[i];
      if (error || !card) {
        continue;
      }
      const minuteKey = GET_MINUTE_KEY_FN(this.connectionKey, batch[i]);
      if ((card as number) <= SMALL_SET_THRESHOLD) {
        if (collect(await redis.smembers(minuteKey))) {
          return names;
        }
        continue;
      }
      // Hot minute (a fast backtest replay packs many records into one
      // wall-clock minute): page through SSCAN with early exit instead of
      // pulling the whole set in a single SMEMBERS
      let cursor: string | number = 0;
      while (true) {
        const [nextCursor, members] = await redis.sscan(
          minuteKey, cursor, "COUNT", SSCAN_BATCH_SIZE
        );
        cursor = nextCursor;
        if (collect(members)) {
          return names;
        }
        if (cursor === "0" || cursor === 0) {
          break;
        }
      }
    }
  }
  return names;
};
The algorithm has three distinct phases per batch of 1 000 minutes:
  1. SCARD probe (one pipeline round trip per 1 000 minutes). Returns the cardinality of each minute SET. Empty minutes (card === 0) are silently skipped — no members are transferred.
  2. SMEMBERS for small sets (≤ 1 000 members). A single command returns the whole set. This is the common case for real-time trading, where at most a few entries land in any given minute.
  3. SSCAN for hot minutes (> 1 000 members). A fast backtest replay can pack thousands of records into a single wall-clock minute. SSCAN pages through the set in batches of 1 000 members with early exit as soon as limit is reached — there is no need to download the entire set before starting to collect names.
Steady-state cost of readLogData at startup is approximately 1 RTT for the floor key + 1–2 pipeline RTTs for cardinality probes + at most limit point GET calls for the object bodies — independent of how many objects the bucket holds in total.

​
Cold-Index Fallback

If Redis is flushed (e.g. container restart with no persistence, or an explicit FLUSHALL), the floor key is gone and listNewest returns an empty slice. LogDataService.listAll detects this and falls back to a direct S3 bucket listing, simultaneously re-warming the Redis index: 
// src/lib/services/data/LogDataService.ts — read path
public listAll = async (): Promise<ILogRow[]> => {
  const rows: ILogRow[] = [];
  const names = await this.logConnectionService.listNewest(LIST_LIMIT);
  if (names.length) {
    for (const name of names) {
      const row = await this.get<ILogRow>(name);
      if (row) {
        rows.push(row);
      }
    }
  } else {
    // Cold index (flushed Redis): object keys embed an inverted timestamp,
    // so plain lexicographic listing is already newest-first — read the
    // newest window and warm the index back up
    for await (const value of this.values("", LIST_LIMIT)) {
      const row = value as ILogRow;
      rows.push(row);
      await this.logConnectionService.register(row.id);
      if (rows.length >= LIST_LIMIT) {
        break;
      }
    }
  }
  rows.sort((a, b) => new Date(b.createDate).getTime() - new Date(a.createDate).getTime());
  return rows.slice(0, LIST_LIMIT);
};
The cold fallback works correctly because log (and notification) object keys use inverted timestamps (Number.MAX_SAFE_INTEGER − ms, zero-padded). Plain lexicographic S3 LIST therefore returns objects in newest-first order without any additional sorting. The fallback reads up to LIST_LIMIT objects and registers each one back into Redis, so the next read is warm again.
Do not flush Redis in a production environment without planning for the cold-start cost. On the first readLogData after a flush, the system falls back to an S3 LIST and re-warms the index by issuing register for every returned object. For a bucket with millions of log entries this can take several seconds and consumes significant S3 LIST API quota. If you must restart Redis, consider preserving the RDB/AOF snapshot, or at minimum pre-warm the index by running a single readLogData call before directing live traffic to the instance.

​
Write Order Guarantee

Every data service that touches both MinIO and Redis writes to MinIO first, then updates the Redis index. The comment in LogDataService.upsert captures the reasoning precisely: 
// src/lib/services/data/LogDataService.ts
// MinIO first (source of truth), Redis index second: a crash in between
// leaves the object readable by key but invisible to listAll — never a phantom
await this.set(key, row);
await this.logConnectionService.register(key);
If the process crashes after the set but before the register, the object exists in MinIO and can be retrieved by key. It will be absent from the Redis index until the cold-index fallback re-discovers it on the next listAll. This is the safest possible failure mode: missing from a listing is recoverable; a phantom Redis entry pointing at a non-existent object would require an explicit cleanup scan. The reverse order — Redis first, MinIO second — would create a window where a Redis entry names an object that does not yet exist, causing get to return null for a name that appears in the listing. That scenario is explicitly avoided by the write order guarantee.

Build docs developers (and LLMs) love

Get started for freeTalk to us