Storage, Notification, Log adapters with Redis time index

Storage, Notification, and Log are the three entities backed by both MinIO as the source of truth and a Redis time-ordered index. The Redis index enables O(limit) reads at startup — only the most recent limit objects are fetched, completely independent of how many objects the bucket holds in total. Notification and Log embed an inverted millisecond timestamp in every object key so that plain S3 lexicographic listing is already newest-first with no runtime sorting. Storage uses a stable backtest/signalId key with no timestamp component — each signal is rewritten in place, and the Redis index is the only mechanism that provides time-ordered listings for Storage.

Inverted timestamps

Notification and Log embed an inverted millisecond timestamp in every object key:

const TIMESTAMP_PAD = String(Number.MAX_SAFE_INTEGER).length; // 16 digits

const inverted = String(Number.MAX_SAFE_INTEGER - when.getTime())
  .padStart(TIMESTAMP_PAD, "0");
// e.g. Date.now() = 1_718_000_000_000  →  inverted = "0007896044036854"

Because MAX_SAFE_INTEGER − t decreases as real time t increases, lexicographically smaller keys are newer. A plain LIST call to MinIO therefore delivers the most recent Notification and Log objects first with zero sort overhead — useful as the cold-index fallback for those two adapters when Redis has been flushed. Storage does not use inverted timestamps; its cold-index fallback performs an unordered full prefix walk of the bucket before re-warming the Redis index.

The write path always registers the object name in the Redis minute-SET index after the MinIO PUT completes. A crash between the two leaves the object readable by key but invisible to listings — never a phantom entry pointing at nothing.

Storage adapter

Purpose

The Storage adapter archives the closed (and open) signal log for a given run mode. Each signal is stored as a JSON object under a stable backtest/signalId key. Because the key is a pure function of context, an upsert is a single idempotent PUT — no read-before-write and no duplicate-key races even when the library replays the whole accumulated list on every write cycle.

Constructor

PersistStorageAdapter.usePersistStorageAdapter(
  class implements IPersistStorageInstance {
    constructor(readonly backtest: boolean) {}
    // ...
  }
);

backtest: boolean is injected by backtest-kit and selects between live and backtest namespaces inside the bucket.

Object key pattern

storage-items/  <backtest>/<signalId>
                 true/abc123
                false/abc123

The key is a stable, mutable object: every signal mutation rewrites the same key. There is no timestamp component, so lexicographic order reflects insertion order only.

Interface methods

Method	Signature	Description
`waitForInit`	`(initial: boolean) => Promise<void>`	Gates first-touch on Redis readiness. No-op on subsequent calls.
`readStorageData`	`() => Promise<StorageData>`	Returns all stored signal rows for the current `backtest` mode.
`writeStorageData`	`(signals: StorageData) => Promise<void>`	Upserts each signal; registers key in Redis index on first write.

// From setup.ts
PersistStorageAdapter.usePersistStorageAdapter(class implements IPersistStorageInstance {
  constructor(readonly backtest: boolean) {}
  async waitForInit(initial: boolean) {
    if (!initial) return;
    await waitForInfra();
  }
  async readStorageData(): Promise<StorageData> {
    const rows = await ioc.storageDataService.listByMode(this.backtest);
    return rows.map((row) => row.payload);
  }
  async writeStorageData(signals: StorageData): Promise<void> {
    for (const signal of signals) {
      await ioc.storageDataService.upsert(this.backtest, signal.id, signal);
    }
  }
});

Redis connection service — `StorageConnectionService`

StorageConnectionService (Redis key prefix: storage-items-connection) maintains one Redis SET per wall-clock minute. On each register(objectName) call it pipelines:

// From StorageConnectionService.ts
const minute = alignToInterval(new Date(), "1m").getTime();
await redis
  .pipeline()
  .sadd(GET_MINUTE_KEY_FN(this.connectionKey, minute), objectName)
  .setnx(GET_FLOOR_KEY_FN(this.connectionKey), String(minute))
  .exec();

SADD deduplicates repeated names within a minute by construction.
SETNX writes the floor marker only once (the earliest minute ever seen), which bounds the backwards walk in listNewest.

Read path

listByMode uses the Redis index for O(limit) access, falling back to MinIO lexicographic listing when the index is cold:

// From StorageDataService.ts
public listByMode = async (backtest: boolean): Promise<IStorageRow[]> => {
  const rows: IStorageRow[] = [];
  const names = await this.storageConnectionService.listNewest(
    Number.POSITIVE_INFINITY,
    `${backtest}/`
  );
  if (names.length) {
    // Warm path: point GETs for each registered name
    for (const name of names) {
      const row = await this.get<IStorageRow>(name);
      if (row) rows.push(row);
    }
  } else {
    // Cold path: full prefix walk (Storage keys have no inverted timestamp,
    // so listing order is lexicographic by signalId, not by time) and re-warm index
    for await (const value of this.values(`${backtest}/`)) {
      const row = value as IStorageRow;
      rows.push(row);
      await this.storageConnectionService.register(row.id);
    }
  }
  return rows;
};

In-process key deduplication

StorageDataService maintains a FIFO-capped Set<string> (REGISTERED_KEYS_LIMIT = 10_000) of already-registered object names. Because backtest-kit re-sends the full accumulated signal list on every write cycle, this set prevents a redundant SADD pipeline call for every signal on every tick — the key is registered only once per process lifetime.

Notification adapter

Purpose

The Notification adapter persists event notifications emitted by strategies (trade opens, closes, alerts, and similar domain events). Each notification is an immutable event: the same notificationId + same createdAt timestamp always maps to identical content, so an existing object is never rewritten.

Constructor

PersistNotificationAdapter.usePersistNotificationAdapter(
  class implements IPersistNotificationInstance {
    constructor(readonly backtest: boolean) {}
    // ...
  }
);

backtest: boolean namespaces notifications by run mode — backtest replays and live runs share the same bucket but never mix entries.

Object key pattern

notification-items/  <backtest>/<⟲ts>_<notificationId>
                      true/0007896044036854_notif-xyz
                     false/0007896044036854_notif-xyz

⟲ts is the inverted millisecond timestamp of notification.createdAt. Lexicographic S3 listing within a backtest/ prefix therefore yields newest notifications first. The key is computed as:

// From NotificationDataService.ts
const TIMESTAMP_PAD = String(Number.MAX_SAFE_INTEGER).length;

const GET_STORAGE_KEY_FN = (
  backtest: boolean,
  notificationId: string,
  when: Date
) => {
  const inverted = String(Number.MAX_SAFE_INTEGER - when.getTime())
    .padStart(TIMESTAMP_PAD, "0");
  return `${backtest}/${inverted}_${notificationId}`;
};

Interface methods

Method	Signature	Description
`waitForInit`	`(initial: boolean) => Promise<void>`	Gates first-touch on Redis readiness.
`readNotificationData`	`() => Promise<NotificationData>`	Returns the newest 200 notifications for the current mode, oldest-first.
`writeNotificationData`	`(notifications: NotificationData) => Promise<void>`	Upserts each notification; skips already-persisted keys.

// From setup.ts
PersistNotificationAdapter.usePersistNotificationAdapter(class implements IPersistNotificationInstance {
  constructor(readonly backtest: boolean) {}
  async waitForInit(initial: boolean) {
    if (!initial) return;
    await waitForInfra();
  }
  async readNotificationData(): Promise<NotificationData> {
    const rows = await ioc.notificationDataService.listByMode(this.backtest);
    return rows.map((row) => row.payload).reverse();
  }
  async writeNotificationData(notifications: NotificationData): Promise<void> {
    for (const notification of notifications) {
      await ioc.notificationDataService.upsert(
        this.backtest,
        notification.id,
        notification
      );
    }
  }
});

The .reverse() call in readNotificationData is intentional and required. Both the Redis index and the MinIO fallback return notifications newest-first (Redis walks backwards from the current minute; MinIO lexicographic order is newest-first via inverted timestamps). The IPersistNotificationInstance contract, however, expects the array oldest-first. The reverse() converts between the two orderings before the data is handed back to backtest-kit.

Redis connection service — `NotificationConnectionService`

NotificationConnectionService (Redis key prefix: notification-items-connection) follows the same minute-SET pattern as StorageConnectionService. The listNewest(limit, prefix) call walks backwards from the current wall-clock minute to the floor, pipelining 1000 minute probes per round trip:

// From NotificationConnectionService.ts
const names = await this.notificationConnectionService.listNewest(
  LIST_LIMIT,      // 200
  `${backtest}/`
);

Read path

// From NotificationDataService.ts
public listByMode = async (backtest: boolean): Promise<INotificationRow[]> => {
  const rows: INotificationRow[] = [];
  const names = await this.notificationConnectionService.listNewest(
    LIST_LIMIT,
    `${backtest}/`
  );
  if (names.length) {
    // Warm path: point GETs for each registered name (newest first)
    for (const name of names) {
      const row = await this.get<INotificationRow>(name);
      if (row) rows.push(row);
    }
  } else {
    // Cold path: bucket listing is already newest-first; re-warm index
    for await (const value of this.values(`${backtest}/`, LIST_LIMIT)) {
      const row = value as INotificationRow;
      rows.push(row);
      await this.notificationConnectionService.register(row.id);
      if (rows.length >= LIST_LIMIT) break;
    }
  }
  rows.sort((a, b) =>
    new Date(b.createDate).getTime() - new Date(a.createDate).getTime()
  );
  return rows.slice(0, LIST_LIMIT);
};

In-process skip of already-persisted keys

NotificationDataService maintains a FIFO-capped Set<string> (PERSISTED_KEYS_LIMIT = 10_000) of object names that have already been written. Because backtest-kit re-sends the full accumulated notification list on every write cycle, this in-process index prevents redundant has() stat calls to MinIO for notifications already known to be persisted.

Log adapter

Purpose

The Log adapter persists strategy log entries — structured messages produced by the framework during signal evaluation, order routing, and risk checks. Log entries are immutable: once written, a (entryId, priority-timestamp) pair never changes content.

Constructor

The Log adapter has no constructor parameters. It is a process-global singleton — all strategies, all run modes, and all threads share one log stream:

PersistLogAdapter.usePersistLogAdapter(class implements IPersistLogInstance {
  async waitForInit(initial: boolean) {
    if (!initial) return;
    await waitForInfra();
  }
  // ...
});

Object key pattern

log-items/  <⟲ts>_<entryId>
             0007896044036854_entry-abc

⟲ts is the inverted millisecond timestamp of entry.priority (the log entry’s wall-clock time). There is no mode prefix — all entries share the same top-level namespace. The key is computed as:

// From LogDataService.ts
const TIMESTAMP_PAD = String(Number.MAX_SAFE_INTEGER).length;

const GET_STORAGE_KEY_FN = (entryId: string, when: Date) => {
  const inverted = String(Number.MAX_SAFE_INTEGER - when.getTime())
    .padStart(TIMESTAMP_PAD, "0");
  return `${inverted}_${entryId}`;
};

Interface methods

Method	Signature	Description
`waitForInit`	`(initial: boolean) => Promise<void>`	Gates first-touch on Redis readiness.
`readLogData`	`() => Promise<LogData>`	Returns the newest 200 log entries, oldest-first.
`writeLogData`	`(entries: LogData) => Promise<void>`	Upserts each entry; skips already-persisted keys.

// From setup.ts
PersistLogAdapter.usePersistLogAdapter(class implements IPersistLogInstance {
  async waitForInit(initial: boolean) {
    if (!initial) return;
    await waitForInfra();
  }
  async readLogData(): Promise<LogData> {
    const rows = await ioc.logDataService.listAll();
    return rows.map((row) => row.payload).reverse();
  }
  async writeLogData(entries: LogData): Promise<void> {
    for (const entry of entries) {
      await ioc.logDataService.upsert(entry.id, entry);
    }
  }
});

Redis connection service — `LogConnectionService`

LogConnectionService (Redis key prefix: log-items-connection) is the reference implementation of the minute-SET pattern. It exposes the same register / listNewest surface as the other two connection services:

// From LogConnectionService.ts — register
const minute = alignToInterval(new Date(), "1m").getTime();
await redis
  .pipeline()
  .sadd(GET_MINUTE_KEY_FN(this.connectionKey, minute), objectName)
  .setnx(GET_FLOOR_KEY_FN(this.connectionKey), String(minute))
  .exec();

listNewest walks backwards from the current minute in batches of 1000, uses SCARD to skip empty minutes without transferring members, and pages hot minutes (where a fast backtest replay packs many entries into one wall-clock minute) via SSCAN with early exit at limit.

Read path

// From LogDataService.ts — listAll (the full read path)
public listAll = async (): Promise<ILogRow[]> => {
  const rows: ILogRow[] = [];
  const names = await this.logConnectionService.listNewest(LIST_LIMIT); // 200
  if (names.length) {
    // Warm path: point GETs for each registered name
    for (const name of names) {
      const row = await this.get<ILogRow>(name);
      if (row) rows.push(row);
    }
  } else {
    // Cold path (flushed Redis): bucket listing is already newest-first
    for await (const value of this.values("", LIST_LIMIT)) {
      const row = value as ILogRow;
      rows.push(row);
      await this.logConnectionService.register(row.id);
      if (rows.length >= LIST_LIMIT) break;
    }
  }
  rows.sort((a, b) =>
    new Date(b.createDate).getTime() - new Date(a.createDate).getTime()
  );
  return rows.slice(0, LIST_LIMIT);
};

Steady-state cost at startup: 1 RTT for the floor key + 1–2 pipeline RTTs for cardinality probes + at most LIST_LIMIT (200) point GETs for bodies — independent of total bucket size.

In-process skip of already-persisted keys

LogDataService maintains a FIFO-capped Set<string> (PERSISTED_KEYS_LIMIT = 10_000) of already-written object names. Because backtest-kit re-sends the full accumulated log list on every write cycle, the in-process index prevents a redundant has() stat call to MinIO for every entry that is already known to be stored.

Shared Redis index mechanics

All three *ConnectionService implementations follow an identical pattern:

Write (register)
Read (listNewest)
Cold fallback

// One Redis SET per wall-clock minute
// Key: <entity>-items-connection:<zero-padded-minute-ms>
await redis
  .pipeline()
  .sadd(minuteKey, objectName)   // idempotent set membership
  .setnx(floorKey, minute)       // write-once lower bound
  .exec();

// Walk backwards from now to floor in batches of 1000 minutes
// 1. Pipeline SCARD for each minute → skip empty minutes free
// 2. SMEMBERS for small sets (≤1000 members)
// 3. SSCAN with early exit for hot minutes (>1000 members)
const names = await connectionService.listNewest(limit, prefix);

// Redis flushed: fall back to MinIO lexicographic listing
// Inverted-timestamp keys are already newest-first
for await (const value of this.values(prefix, limit)) {
  rows.push(value);
  await connectionService.register(value.id); // re-warm index
}

Get Started

Architecture

Configuration

Persist Adapters

Strategy Development

Storage, Notification, Log adapters with Redis time index

Inverted timestamps

Storage adapter

Purpose

Constructor

Object key pattern

Interface methods

Redis connection service — `StorageConnectionService`

Read path

Notification adapter

Purpose

Constructor

Object key pattern

Interface methods

Redis connection service — `NotificationConnectionService`

Read path

Log adapter

Purpose

Constructor

Object key pattern

Interface methods

Redis connection service — `LogConnectionService`

Read path

Shared Redis index mechanics

Build docs developers (and LLMs) love

Get Started

Architecture

Configuration

Persist Adapters

Strategy Development

Documentation Index

​Inverted timestamps

​Storage adapter

​Purpose

​Constructor

​Object key pattern

​Interface methods

​Redis connection service — StorageConnectionService

​Read path

​Notification adapter

​Purpose

​Constructor

​Object key pattern

​Interface methods

​Redis connection service — NotificationConnectionService

​Read path

​Log adapter

​Purpose

​Constructor

​Object key pattern

​Interface methods

​Redis connection service — LogConnectionService

​Read path

​Shared Redis index mechanics

Build docs developers (and LLMs) love

Inverted timestamps

Storage adapter

Purpose

Constructor

Object key pattern

Interface methods

Redis connection service — `StorageConnectionService`

Read path

Notification adapter

Purpose

Constructor

Object key pattern

Interface methods

Redis connection service — `NotificationConnectionService`

Read path

Log adapter

Purpose

Constructor

Object key pattern

Interface methods

Redis connection service — `LogConnectionService`

Read path

Shared Redis index mechanics