Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/backtest-kit/backtest-kit-minio-s3-docker/llms.txt

Use this file to discover all available pages before exploring further.

backtest-kit-minio-s3-docker replaces the default file-based ./dump/ persistence with a deliberate two-layer design: MinIO (S3) is the source of truth for every entity, while Redis acts as a fast time-ordered index for the three entities that need newest-first listings. Strategy code, runners, and the CLI entrypoint are completely unchanged — only the persistence layer is swapped via 16 custom IPersist*Instance adapters registered in src/config/setup.ts. The two layers have different jobs and are never interchangeable: MinIO owns durability, Redis owns order.

​
The Persistence Contract

backtest-kit imposes a strict write durability contract on every persistence adapter: after writeXData(...) returns, the very next readXData(...) call must observe the just-written value. No eventual consistency, no cache warm-up window. S3 (and MinIO’s S3-compatible API) provides strong read-after-write consistency for individual objects: a successful PUT is immediately visible to any subsequent GET or HEAD on the same key. Because every entity’s object key is a deterministic function of its context fields, an upsert maps to a single idempotent PUT with no read-before-write or duplicate-key races. The contract therefore holds with plain object semantics — no distributed transactions required.

​
Write Durability

Four properties together provide database-level durability guarantees without a database:
  1. Deterministic keys. Every record’s object key is a pure function of its context (symbol/strategy/exchange/…). An upsert is a single idempotent PUT — calling it twice produces exactly the same object with the same key.
  2. Immutable entities never rewrite. Candles use a stat + PUT insert-only pair: if the key already exists the write is skipped entirely, since the stored body is fully determined by the DTO. Log entries and notifications also skip the PUT when the key already exists, and an in-process FIFO-capped index of persisted keys (PERSISTED_KEYS_LIMIT = 10_000) prevents redundant network stat calls in steady state.
  3. Write order: MinIO first, Redis second. Every service that touches both layers writes the S3 object before registering in the Redis index. A crash between the two leaves an object that is readable by key but temporarily invisible to listings — never a phantom Redis entry pointing at a missing object.
  4. removed means absent. Soft-delete entities (Measure, Interval, Memory) physically delete the S3 object rather than writing a tombstone flag. listKeys is then a pure prefix LIST with zero body reads, and reads of removed entries return null by construction.
Crash safety flows directly from write order: because MinIO is written first, a process killed between the MinIO write and the Redis SADD leaves the system in a safe state. The object exists and is readable by key. The Redis index will be missing that one entry, but it will be recovered automatically the next time listNewest falls back to the cold-index path and re-warms Redis from the bucket listing.

​
Data Flow

The following diagram traces a single writeLogData call through both layers: 
writeLogData(entries)
│
├─ 1. Adapter iterates entries
│     (src/config/setup.ts → PersistLogAdapter)
│
├─ 2. logDataService.upsert(entryId, entry)
│     Compute inverted-timestamp key:
│     key = `${MAX_SAFE_INTEGER - when.getTime()}_${entryId}`
│
├─ 3. Check in-process _persistedKeys cache
│     └─ Hit → return immediately (zero network)
│
├─ 4. BaseStorage.has(key)  ←  MinIO statObject
│     └─ Exists → _rememberKey(key), return
│
├─ 5. BaseStorage.set(key, row)  ←  MinIO putObject  ✓ SOURCE OF TRUTH
│
├─ 6. logConnectionService.register(key)
│     minute = alignToInterval(now, "1m")
│     Redis pipeline:
│       SADD  log-items-connection:<minute>  key
│       SETNX log-items-connection:floor    <minute>
│                                           ✓ TIME INDEX UPDATED
│
└─ 7. _rememberKey(key)  ←  in-process dedup cache updated
A readLogData call follows the reverse path: 
readLogData()
│
├─ 1. logConnectionService.listNewest(200)
│     └─ Redis floor key present?
│         ├─ Yes → backwards walk (see redis-index page)
│         └─ No  → cold-index fallback: S3 LIST (newest-first by key)
│                  + re-warm Redis index
│
└─ 2. For each name: BaseStorage.get(name)  ←  MinIO getObject

​
All 16 Entities

Every adapter lives in one MinIO bucket called backtest-kit. Each entity occupies its own root folder. The three entities marked † also maintain a Redis time index.
AdapterFolderObject Key FormatPurpose
Candlecandle-items/exchange/symbol/interval/timestampOHLCV cache; immutable inserts
Signalsignal-items/symbol/strategy/exchangeLive signal state per context
Scheduleschedule-items/symbol/strategy/exchangePending scheduled signal
Strategystrategy-items/symbol/strategy/exchangeDeferred commit queue snapshot
Riskrisk-items/riskName/exchangeActive risk positions snapshot
Partialpartial-items/symbol/strategy/exchange/signalIdPartial profit/loss levels per signal
Breakevenbreakeven-items/symbol/strategy/exchange/signalIdBreakeven reached flag
Storage †storage-items/backtest/signalIdClosed/opened signal log per mode
Notification †notification-items/backtest/⟲ts_notificationIdEvent notifications
Log †log-items/⟲ts_entryIdStrategy log entries
Measuremeasure-items/bucket/entryKeyLLM/API response cache
Intervalinterval-items/bucket/entryKeyOnce-per-interval markers
Memorymemory-items/signalId/bucket/memoryIdPer-signal memory store
Recentrecent-items/symbol/strategy/exchange/frame/backtestLast public signal per context
Statestate-items/signalId/bucketPer-signal state buckets
Sessionsession-items/strategy/exchange/frame/symbol/backtestOne session per running strategy
⟲ts denotes an inverted timestamp (Number.MAX_SAFE_INTEGER − ms, zero-padded to a fixed width). Plain lexicographic S3 listing yields newest-first with no additional sorting, which is exactly what the cold-index fallback relies on.

​
Explore the Two Layers

MinIO Storage

Bucket layout, object key schemas, BaseStorage factory operations, and the candle insert-only pattern.

Redis Index

Minute-bucket design, the listNewest backwards-walk algorithm, cold-index fallback, and write-order guarantees.

Build docs developers (and LLMs) love

Get started for freeTalk to us