Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/backtest-kit/backtest-kit-minio-s3-docker/llms.txt

Use this file to discover all available pages before exploring further.

Some entities need newest-first listings that S3 lexicographic ordering alone cannot provide efficiently. Log, Notification, and Storage solve this by maintaining a Redis minute-bucket index alongside their MinIO objects. Each write registers the new object key in a Redis SET partitioned by aligned minute; reads walk the index backwards from the current minute, fetching only as many keys as needed — independent of how many objects the bucket holds in total. The remaining three adapters — Measure, Interval, and Memory — are utility caches that use physical deletion instead of tombstones, keeping listKeys a pure prefix LIST with zero body reads.

​
Log

The Log adapter persists strategy log entries emitted by backtest-kit. There is no per-context constructor argument — all log entries share a single global namespace within the log-items/ prefix. Constructor 
// No constructor arguments — the Log adapter is a singleton per process.
async waitForInit(initial: boolean): Promise<void>
Read method 
async readLogData(): Promise<LogData>
Returns the newest 200 log entries. Internally, LogDataService.listAll() queries the Redis time index first via logConnectionService.listNewest(200), then issues one point GET per returned key. If the Redis index is empty (cold start after a flush), the method falls back to a lexicographic bucket listing — which is already newest-first thanks to the inverted timestamp key — and warms the index back up: 
// src/lib/services/data/LogDataService.ts — read path
const names = await this.logConnectionService.listNewest(LIST_LIMIT);
if (names.length) {
  for (const name of names) {
    const row = await this.get<ILogRow>(name);
    if (row) rows.push(row);
  }
} else {
  // Cold index (flushed Redis): object keys embed an inverted timestamp,
  // so plain lexicographic listing is already newest-first — read the
  // newest window and warm the index back up
  for await (const value of this.values("", LIST_LIMIT)) {
    const row = value as ILogRow;
    rows.push(row);
    await this.logConnectionService.register(row.id);
    if (rows.length >= LIST_LIMIT) break;
  }
}
// Final sort and cap — normalises order regardless of which path ran
rows.sort((a, b) => new Date(b.createDate).getTime() - new Date(a.createDate).getTime());
return rows.slice(0, LIST_LIMIT);
Write method 
async writeLogData(entries: LogData): Promise<void>
Iterates over the supplied entries and calls logDataService.upsert(entry.id, entry) for each one. upsert uses an in-process _persistedKeys set to skip entries that have already been written this process lifetime — no network round-trip needed. For entries not in the set, a has() stat checks MinIO existence before writing. The write order is MinIO first, Redis second: a crash between the two leaves the object readable by direct key lookup but invisible to listAll — never a phantom index entry pointing at nothing. Object key — ⟲ts_entryId The inverted timestamp is zero-padded to a fixed width so that lexicographic order equals reverse-chronological order: 
const TIMESTAMP_PAD = String(Number.MAX_SAFE_INTEGER).length;

const GET_STORAGE_KEY_FN = (entryId: string, when: Date) => {
  const inverted = String(Number.MAX_SAFE_INTEGER - when.getTime())
    .padStart(TIMESTAMP_PAD, "0");
  return `${inverted}_${entryId}`;
};
LogDataService maintains an in-process _persistedKeys Set capped at 10,000 entries (FIFO eviction). backtest-kit re-sends the entire accumulated log list on every writeLogData call. Without this set, each call would cost one has() stat per entry against MinIO. The in-process index reduces steady-state write cost to zero network round-trips for entries already written in this process lifetime.

​
Notification

The Notification adapter persists event notifications — trade executions, alerts, system messages — with the same Redis-backed newest-first read pattern as Log. The backtest flag partitions live and backtest notifications into separate key prefixes. Constructor 
constructor(readonly backtest: boolean) {}
Read method 
async readNotificationData(): Promise<NotificationData>
Calls notificationDataService.listByMode(this.backtest), which queries the Redis index with listNewest(200, "${backtest}/"). The service sorts the returned rows by createDate descending and slices to 200. The adapter then calls .reverse() on the result before returning — backtest-kit expects the array in oldest-first order for replay: 
async readNotificationData(): Promise<NotificationData> {
  const rows = await ioc.notificationDataService.listByMode(this.backtest);
  return rows.map((row) => row.payload).reverse();
}
Write method 
async writeNotificationData(notifications: NotificationData): Promise<void>
Iterates over the supplied notifications, computing the inverted-timestamp key from each notification’s createdAt field. Like Log, it maintains an in-process _persistedKeys set (capped at 10,000) to skip already-written entries without a MinIO stat. Object key — <backtest>/⟲ts_notificationId 
const GET_STORAGE_KEY_FN = (
  backtest: boolean,
  notificationId: string,
  when: Date
) => {
  const inverted = String(Number.MAX_SAFE_INTEGER - when.getTime())
    .padStart(TIMESTAMP_PAD, "0");
  return `${backtest}/${inverted}_${notificationId}`;
};
If Redis is flushed while notifications are accumulating, the cold-index fallback for Notification (and Log) uses the lexicographic bucket listing, which is already newest-first because of the inverted timestamp. The fallback also re-registers each object key in the Redis index, so subsequent reads return to the fast index path automatically — no manual intervention needed.

​
Storage

The Storage adapter persists the closed and opened signal log — a record of every signal that has passed through the strategy, partitioned by mode (backtest: true or false). Unlike Log and Notification, signal objects are mutable: a signal is rewritten on every state transition under its stable key. The Redis index tracks which keys exist, not their current content. Constructor 
constructor(readonly backtest: boolean) {}
Read method 
async readStorageData(): Promise<StorageData>
Calls storageDataService.listByMode(this.backtest), which queries the Redis index with listNewest(Infinity, "${backtest}/") to retrieve all signal keys for the current mode, then fetches each object body. Cold-index fallback walks the bucket prefix "${backtest}/" directly and warms Redis: 
// Cold index fallback in StorageDataService
for await (const value of this.values(`${backtest}/`)) {
  const row = value as IStorageRow;
  rows.push(row);
  await this.storageConnectionService.register(row.id);
}
Write method 
async writeStorageData(signals: StorageData): Promise<void>
Iterates over the supplied signals and calls storageDataService.upsert(this.backtest, signal.id, signal) for each. Every upsert is a full overwrite of the MinIO object under the stable key, followed by a one-time Redis registration per key (guarded by an in-process _registeredKeys set capped at 10,000). Object key — <backtest>/<signalId> 
const GET_STORAGE_KEY_FN = (backtest: boolean, signalId: string) =>
  `${backtest}/${signalId}`;

​
Utility Adapters: Measure, Interval, Memory

These three adapters implement soft-delete semantics by physically deleting the MinIO object when an entry is removed. There are no tombstone markers to skip during listing — listKeys is a pure prefix LIST that returns only live entries, with zero body reads.

​
Measure

The Measure adapter is a keyed cache for LLM and external API responses. It is scoped to a bucket string that groups related entries. Constructor 
constructor(readonly bucket: string) {}
Key methods 
async readMeasureData(key: string): Promise<MeasureData | null>
async writeMeasureData(data: MeasureData, key: string, _when: Date): Promise<void>
async removeMeasureData(key: string): Promise<void>
async *listMeasureData(): AsyncGenerator<string>
Object key schema — measure-items/<bucket>/<entryKey> 
const GET_STORAGE_KEY_FN = (bucket: string, entryKey: string) =>
  `${bucket}/${entryKey}`;
listMeasureData() is an async generator that yields each entryKey string (with the bucket/ prefix stripped) by issuing a single keys(prefix) LIST against MinIO: 
async *listMeasureData(): AsyncGenerator<string> {
  const keys = await ioc.measureDataService.listKeys(this.bucket);
  for (const key of keys) {
    yield key;
  }
}
removeMeasureData physically deletes the object via softRemove, which calls this.delete(key) on the underlying BaseStorage. The removed flag on MeasureData is also checked in upsert: if payload.removed === true, the upsert path itself calls delete instead of set.

​
Interval

The Interval adapter stores once-per-interval markers — flags that record whether a periodic action has already been taken within a given interval window. Like Measure, it is scoped to a bucket. Constructor 
constructor(readonly bucket: string) {}
Key methods 
async readIntervalData(key: string): Promise<IntervalData | null>
async writeIntervalData(data: IntervalData, key: string, when: Date): Promise<void>
async removeIntervalData(key: string): Promise<void>
async *listIntervalData(): AsyncGenerator<string>
Object key schema — interval-items/<bucket>/<entryKey> 
const GET_STORAGE_KEY_FN = (bucket: string, entryKey: string) =>
  `${bucket}/${entryKey}`;
The structure and semantics are identical to Measure — same soft-delete via physical deletion, same pure-LIST listIntervalData() generator.

​
Memory

The Memory adapter provides a per-signal memory store used for BM25 index rebuilds and other signal-scoped caches. It is doubly scoped by signalId and bucketName, and each entry is additionally identified by a memoryId. Constructor 
constructor(
  readonly signalId:    string,
  readonly bucketName:  string,
) {}
Key methods 
async readMemoryData(memoryId: string): Promise<MemoryData | null>
async hasMemoryData(memoryId: string): Promise<boolean>
async writeMemoryData(data: MemoryData, memoryId: string, when: Date): Promise<void>
async removeMemoryData(memoryId: string): Promise<void>
async *listMemoryData(): AsyncGenerator<{ memoryId: string; data: MemoryData }>
dispose(): void
Object key schema — memory-items/<signalId>/<bucket>/<memoryId> 
const GET_STORAGE_KEY_FN = (
  signalId: string,
  bucketName: string,
  memoryId: string
) => `${signalId}/${bucketName}/${memoryId}`;
listMemoryData() yields { memoryId, data } pairs. Unlike Measure and Interval (which only need keys), Memory fetches full object bodies because callers need the content for index reconstruction: 
async *listMemoryData(): AsyncGenerator<{ memoryId: string; data: MemoryData }> {
  const rows = await ioc.memoryDataService.listEntries(this.signalId, this.bucketName);
  for (const row of rows) {
    yield { memoryId: row.memoryId, data: row.payload };
  }
}
dispose() is a no-op (void 0) — per-signal memory cleanup is managed by backtest-kit at the framework level, not by the adapter.

Build docs developers (and LLMs) love

Get started for freeTalk to us