backtest-kit MinIO S3 Docker: persistence layer overview

backtest-kit-minio-s3-docker is an integration of backtest-kit that swaps the default file-based ./dump/ persistence for a production-grade, two-container stack: MinIO (S3) as the source of truth for all object storage and Redis as a time-ordered index for newest-first listings. The project ships 16 custom IPersist*Instance adapters that implement the full backtest-kit persistence contract on top of that stack, packaged with Docker Compose for one-command deploys. Strategy code, runners, and the CLI entrypoint are completely unchanged — only the persistence layer is replaced.

Quickstart

Stand up MinIO and Redis, configure credentials, build your strategy bundle, and run your first backtest in under 5 minutes.

Architecture Overview

Dive into how the 16 adapters are wired, how the bucket layout is organised, and how the Redis minute-index works.

Adapters Overview

Explore every adapter — Candle, Signal, Log, Storage, Notification, and more — with their S3 key schemas and durability guarantees.

Environment Configuration

Reference every environment variable consumed by the container and the Node.js host runner, from MinIO credentials to CLI flags.

Why MinIO + Redis instead of plain files?

The default backtest-kit file adapter writes JSON to a local ./dump/ directory — perfect for laptop runs and CI. As soon as you want remote durability, versioning, multi-host access, or an archiveable snapshot of every candle and signal, a plain filesystem becomes a liability. A full relational or document database solves those problems but adds schema management, migration tooling, and a b-tree engine that is overkill when your workload is bounded JSON blobs keyed by trading context. This project occupies the deliberate middle ground: S3-grade durability with zero schema management. MinIO is API-compatible with AWS S3, so you get object versioning, lifecycle rules, mc mirror replication, and cross-region backups without touching a database. Redis provides the thin time-ordered index that S3 lexicographic listing cannot: for the three entities that need “newest first” queries — Log, Notification, and Storage — Redis minute-buckets let listNewest(limit) resolve in 1–2 pipeline RTTs regardless of how many objects the bucket holds.

Choosing the right persistence tier

	Default file `./dump/`	MinIO + Redis (this project)	`@backtest-kit/mongo` / `@backtest-kit/pg`
Infrastructure	none	2 containers	database + Redis cache
Source of truth	JSON files on local disk	JSON objects in S3 bucket	rows / documents
Durability & ops	single host, manual backup	S3 semantics: versioning, replication, `mc mirror`, lifecycle	DB tooling: dumps, replicas
Newest-first listings	directory scan	Redis minute-index, O(limit)	`ORDER BY … LIMIT`, O(log n)
Point reads (candles)	`fs.readFile`	1 GET ≈ 1–3 ms	b-tree lookup ≈ 0.1–1 ms
Sweet spot	local runs, CI	fat JSON snapshots, cheap unbounded archive, S3-native infra	hundreds of millions of candles, ad-hoc SQL/aggregation

When to use this project vs @backtest-kit/pg / @backtest-kit/mongo: choose backtest-kit-minio-s3-docker when you want S3-grade object durability, you operate in an S3-native environment (MinIO on-prem, AWS S3, or compatible), and your total candle set is measured in millions rather than hundreds of millions. If your dataset grows into the hundreds of millions of candles and you need ad-hoc aggregations or SQL joins, switch to the pg or mongo package — b-trees win that workload decisively.

The 16 persist adapters

Each of the 16 adapters implements the corresponding IPersist*Instance interface from backtest-kit and is registered in src/config/setup.ts. Everything lives in one MinIO bucket named backtest-kit — each entity gets its own root folder inside that bucket.

Adapter	Folder	Object key	Purpose
Candle	`candle-items/`	`exchange/symbol/interval/timestamp`	OHLCV cache; immutable inserts
Signal	`signal-items/`	`symbol/strategy/exchange`	Live signal state per context
Schedule	`schedule-items/`	`symbol/strategy/exchange`	Pending scheduled signal
Strategy	`strategy-items/`	`symbol/strategy/exchange`	Deferred commit queue snapshot
Risk	`risk-items/`	`riskName/exchange`	Active risk positions snapshot
Partial	`partial-items/`	`symbol/strategy/exchange/signalId`	Partial profit/loss levels per signal
Breakeven	`breakeven-items/`	`symbol/strategy/exchange/signalId`	Breakeven reached flag
Storage	`storage-items/`	`backtest/signalId`	Closed/opened signal log per mode †
Notification	`notification-items/`	`backtest/⟲ts_notificationId`	Event notifications †
Log	`log-items/`	`⟲ts_entryId`	Strategy log entries †
Measure	`measure-items/`	`bucket/entryKey`	LLM/API response cache
Interval	`interval-items/`	`bucket/entryKey`	Once-per-interval markers
Memory	`memory-items/`	`signalId/bucket/memoryId`	Per-signal memory store
Recent	`recent-items/`	`symbol/strategy/exchange/frame/backtest`	Last public signal per context
State	`state-items/`	`signalId/bucket`	Per-signal state buckets
Session	`session-items/`	`strategy/exchange/frame/symbol/backtest`	One session per running strategy

⟲ts = inverted timestamp (MAX_SAFE_INTEGER − ms, zero-padded). Plain lexicographic S3 listing therefore yields newest first with no sorting overhead. Entities marked † are also maintained in the Redis time index (see below).

The `backtest-kit` bucket layout

The BaseStorage("backtest-kit/<entity>-items") naming convention is parsed as bucket/parent-folder: the first path segment becomes the physical MinIO bucket, and the rest becomes a transparent key prefix. This means all 16 adapters share a single backtest-kit bucket while remaining fully isolated by prefix — no cross-entity key collisions, and MinIO lifecycle rules can be scoped to individual prefixes.

backtest-kit/          ← single MinIO bucket
  candle-items/        ← OHLCV data, immutable
  signal-items/        ← live signal state
  log-items/           ← strategy logs (Redis-indexed)
  notification-items/  ← event notifications (Redis-indexed)
  storage-items/       ← closed/opened signals (Redis-indexed)
  ...                  ← 11 further entity prefixes

Why Redis is needed for time ordering

S3 lists objects in lexicographic key order only — it cannot answer “what are the 200 most recent log entries?” without walking the entire prefix. For the three entities that require newest-first listings (Log, Notification, Storage), this project maintains a Redis minute-bucket index:

One Redis SET per minute: <entity>-connection:<aligned-minute> → set of object names.
register() is a single pipeline (SADD + SETNX floor marker). Timestamps are minute-aligned, so re-registering within a minute deduplicates automatically.
listNewest(limit, prefix) walks backwards from the current minute in pipeline batches of 1,000 probes. A cheap SCARD pass skips empty minutes without transferring members; hot minutes are paged via SSCAN with early exit at limit.
Cold-index fallback: if Redis is flushed, listings fall back to the bucket LIST (inverted-timestamp keys are already newest-first) and re-warm the index in the same pass.

This design keeps the steady-state read cost of readLogData at startup to 1 RTT for the floor marker plus 1–2 pipeline RTTs plus ≤ 200 point GETs for bodies — completely independent of bucket size.

Write durability without a database

backtest-kit’s write durability contract requires that a value written by writeXData(...) must be visible to the very next readXData(...). This project satisfies the contract without transactions:

Deterministic keys. Every object key is a pure function of its trading context (symbol/strategy/exchange/…), so an upsert is a single idempotent PUT — no read-before-write races.
Immutable entities never rewrite. Candles use a stat + PUT insert-only pair backed by an in-process FIFO index of already-persisted keys, so re-sending an accumulated list costs zero network in steady state.
Write order: MinIO first, Redis second. A crash between the two leaves an object readable by key but invisible to listings — never a phantom listing entry pointing at a missing object.
removed means absent. Soft-delete entities physically delete the object; listKeys is a pure prefix LIST with zero body reads, and reads of removed entries return null by construction.

Get Started

Architecture

Configuration

Persist Adapters

Strategy Development

backtest-kit MinIO S3 Docker: persistence layer overview

Quickstart

Architecture Overview

Adapters Overview

Environment Configuration

Why MinIO + Redis instead of plain files?

Choosing the right persistence tier

The 16 persist adapters

The `backtest-kit` bucket layout

Why Redis is needed for time ordering

Write durability without a database

Build docs developers (and LLMs) love

Get Started

Architecture

Configuration

Persist Adapters

Strategy Development

Documentation Index

Quickstart

Architecture Overview

Adapters Overview

Environment Configuration

​Why MinIO + Redis instead of plain files?

​Choosing the right persistence tier

​The 16 persist adapters

​The backtest-kit bucket layout

​Why Redis is needed for time ordering

​Write durability without a database

Build docs developers (and LLMs) love

Why MinIO + Redis instead of plain files?

Choosing the right persistence tier

The 16 persist adapters

The `backtest-kit` bucket layout

Why Redis is needed for time ordering

Write durability without a database