Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/theonetrade/backtest-kit-minio-s3-docker/llms.txt

Use this file to discover all available pages before exploring further.

backtest-kit MinIO S3 Docker replaces the default ./dump/ file-based persistence with a two-tier infrastructure layer: MinIO serves as the S3-compatible source of truth for every JSON record, and Redis provides a per-minute SET index that makes newest-first listings O(limit) instead of O(bucket). Sixteen IPersist*Instance adapters wire this infrastructure into the standard backtest-kit persistence contract, meaning strategy code, runners, and the CLI entry point require zero changes — only the persistence layer is swapped. No schema migrations, no ORM, no SQL: every record is a JSON object stored at a deterministic key path. MinIO’s S3 semantics (versioning, mc mirror, lifecycle policies) handle operational durability; Redis handles time-ordered read efficiency.

Three-Layer Model

LayerWhat it doesWhere it lives
backtest-kit IPersist contractDefines readXData / writeXData interfaces that strategy code callsbacktest-kit npm package
16 IPersist adaptersImplement each interface; delegate to the appropriate data servicesrc/config/setup.ts
MinIO + RedisMinIO stores every JSON object; Redis indexes Log, Notification, and Storage entries for newest-first listingDocker Compose services
Every adapter follows the same skeleton registered in src/config/setup.ts:
src/config/setup.ts
PersistCandleAdapter.usePersistCandleAdapter(class implements IPersistCandleInstance {
  constructor(
    readonly symbol: string,
    readonly interval: CandleInterval,
    readonly exchangeName: string,
  ) {}
  async waitForInit(initial: boolean) {
    if (!initial) return;
    await waitForInfra();   // gates first-touch on Redis ready; MinIO connects lazily
  }
  async writeCandlesData(candles: CandleData[]): Promise<void> {
    for (const candle of candles) {
      await ioc.candleDataService.create({
        symbol: this.symbol,
        interval: this.interval,
        open: candle.open,
        high: candle.high,
        low: candle.low,
        close: candle.close,
        timestamp: candle.timestamp,
        volume: candle.volume,
      });
    }
  }
  async readCandlesData(limit: number, sinceTimestamp: number) {
    // walks per-timestamp MinIO GETs using findBySymbolIntervalTimestamp
  }
});

The 16 Persist Adapters

All adapters share a single MinIO bucket named backtest-kit. Each entity occupies its own root folder within that bucket.
AdapterFolderObject key patternPurpose
Candlecandle-items/exchange/symbol/interval/timestampOHLCV cache; immutable inserts
Signalsignal-items/symbol/strategy/exchangeLive signal state per context
Scheduleschedule-items/symbol/strategy/exchangePending scheduled signal
Strategystrategy-items/symbol/strategy/exchangeDeferred commit queue snapshot
Riskrisk-items/riskName/exchangeActive risk positions snapshot
Partialpartial-items/symbol/strategy/exchange/signalIdPartial profit/loss levels per signal
Breakevenbreakeven-items/symbol/strategy/exchange/signalIdBreakeven reached flag
Storagestorage-items/backtest/signalIdClosed/opened signal log per mode †
Notificationnotification-items/backtest/⟲ts_notificationIdEvent notifications †
Loglog-items/⟲ts_entryIdStrategy log entries †
Measuremeasure-items/bucket/entryKeyLLM/API response cache
Intervalinterval-items/bucket/entryKeyOnce-per-interval markers
Memorymemory-items/signalId/bucket/memoryIdPer-signal memory store
Recentrecent-items/symbol/strategy/exchange/frame/backtestLast public signal per context
Statestate-items/signalId/bucketPer-signal state buckets
Sessionsession-items/strategy/exchange/frame/symbol/backtestOne session per running strategy
Log, Notification, and Storage entries are also registered in the Redis time index on every write. All other adapters read directly from MinIO with no Redis involvement.

Single-Bucket Model and Key Prefixes

Everything lives in one physical MinIO bucket called backtest-kit. BaseStorage is constructed with a name that encodes both the bucket and the entity folder:
BaseStorage.ts
constructor(public readonly BUCKET_NAME: string) {
  const [bucketName, ...folders] = BUCKET_NAME.split("/");
  this.bucketName = bucketName;         // "backtest-kit"
  this.rootPrefix = folders.length
    ? `${folders.join("/")}/`           // "candle-items/"
    : "";
}
For example, BaseStorage("backtest-kit/candle-items") resolves to:
  • Physical bucket: backtest-kit
  • Root key prefix: candle-items/
  • Full object path: backtest-kit/candle-items/ccxt_binance/BTCUSDT/1h/1700000000000
A name without a slash — BaseStorage("breakeven-items") — still means a dedicated bucket named breakeven-items with no prefix, which keeps the implementation fully backward compatible with single-bucket setups.

Inverted Timestamp Keys

For entities that need newest-first ordering (Log, Notification), the object key embeds an inverted timestamp instead of a raw Unix millisecond value:
LogDataService.ts
const TIMESTAMP_PAD = String(Number.MAX_SAFE_INTEGER).length;

const GET_STORAGE_KEY_FN = (entryId: string, when: Date) => {
  const inverted = String(Number.MAX_SAFE_INTEGER - when.getTime())
    .padStart(TIMESTAMP_PAD, "0");
  return `${inverted}_${entryId}`;
};
Because Number.MAX_SAFE_INTEGER (9007199254740991) minus a recent Unix millisecond produces a very large but steadily decreasing number, plain lexicographic S3 listing already returns objects newest-first — no sort pass required. This is the foundation of the cold-index fallback: even with Redis flushed, listObjectsV2 returns entries in the right order.

Explore Further

MinIO Storage

Bucket layout, BaseStorage API, auto-bucket creation, and candle insert-only semantics.

Redis Index

Per-minute SETs, backwards walk, cold-index fallback, and steady-state RTT cost.

Write Durability

How S3 read-after-write consistency and deterministic keys satisfy the backtest-kit write durability contract.

Adapters Overview

All 16 adapters, their registration pattern, and the waitForInit gate.

Build docs developers (and LLMs) love