backtest-kit-minio-s3-docker is a drop-in persistence layer for the backtest-kit algo-trading framework that replaces the default file-basedDocumentation Index
Fetch the complete documentation index at: https://mintlify.com/backtest-kit/backtest-kit-minio-s3-docker/llms.txt
Use this file to discover all available pages before exploring further.
./dump/ storage with MinIO (S3) as the source of truth and Redis as a time-ordered index, packaged with docker-compose for one-command infrastructure deploys. It ships 16 custom IPersist*Instance adapters that fulfill the full backtest-kit persistence contract — strategy code, runners, and the CLI entrypoint are completely unchanged; only the storage layer is swapped out beneath them.
The Three Components
The project composes three layers that each play a distinct role:- backtest-kit — the core algo-trading framework that defines the
IPersist*Instancecontract, theBacktest,Live, andPaperrunners, and the@backtest-kit/clientrypoint. None of its internals are modified. - MinIO (S3) — serves as the primary source of truth. Every persisted entity is stored as a JSON object inside a single
backtest-kitbucket. MinIO provides strong read-after-write consistency for single objects, S3-compatible tooling (mc mirror, lifecycle rules, versioning), and an administrative web console. It exposes the S3 API on port9000and a management UI on port9001. - Redis — maintains a time-ordered index for the three entities that require newest-first listings:
Log,Notification, andStorage. Rather than walking an entire S3 bucket, Redis stores oneSETper minute (keyed by aligned timestamp), allowinglistNewest(limit)to walk backwards in time with a handful of pipeline RTTs — completely independent of how many total objects the bucket contains.
Choosing the Right Persistence Variant
Not every project needs the same persistence backend. The table below shows the practical trade-offs between the three available options:Default file ./dump/ | MinIO + Redis (this project) | @backtest-kit/mongo / @backtest-kit/pg | |
|---|---|---|---|
| Infrastructure | None | 2 containers | Database + Redis cache |
| Source of truth | JSON files on local disk | JSON objects in S3 bucket | Rows / documents |
| Durability & ops | Single host, manual backup | S3 semantics: versioning, replication, mc mirror, lifecycle | DB tooling: dumps, replicas |
| Newest-first listings | Directory scan | Redis minute-index, O(limit) | ORDER BY … LIMIT, O(log n) |
| Point reads (candles) | fs.readFile | 1 GET ≈ 1–3 ms | B-tree lookup ≈ 0.1–1 ms |
| Sweet spot | Local runs, CI | Fat JSON snapshots, cheap unbounded archive, S3-native infra | Hundreds of millions of candles, ad-hoc SQL/aggregation |
mc mirror for off-site replication, or you prefer keeping everything in flat JSON without managing migrations.
If your candle dataset grows into the hundreds of millions of rows, the
@backtest-kit/pg or @backtest-kit/mongo packages are the better choice. B-tree index lookups and ORDER BY … LIMIT queries outperform S3 prefix listing at that scale, and both packages include Redis caching to accelerate repeated reads.The 16 Persist Adapters
The project ships one adapter for eachIPersist*Instance interface defined by backtest-kit. All 16 live inside a single MinIO bucket (backtest-kit), each assigned its own root folder. Adapters that require newest-first ordering additionally maintain a Redis minute-index via a corresponding *ConnectionService.
| Adapter | Folder | Purpose |
|---|---|---|
| Candle | candle-items/ | OHLCV cache; immutable inserts |
| Signal | signal-items/ | Live signal state per context |
| Schedule | schedule-items/ | Pending scheduled signal |
| Strategy | strategy-items/ | Deferred commit queue snapshot |
| Risk | risk-items/ | Active risk positions snapshot |
| Partial | partial-items/ | Partial profit/loss levels per signal |
| Breakeven | breakeven-items/ | Breakeven reached flag |
| Storage | storage-items/ | Closed/opened signal log per mode |
| Notification | notification-items/ | Event notifications (Redis-indexed) |
| Log | log-items/ | Strategy log entries (Redis-indexed) |
| Measure | measure-items/ | LLM/API response cache |
| Interval | interval-items/ | Once-per-interval markers |
| Memory | memory-items/ | Per-signal memory store |
| Recent | recent-items/ | Last public signal per context |
| State | state-items/ | Per-signal state buckets |
| Session | session-items/ | One session per running strategy |
Navigation
Quickstart
Start MinIO and Redis with a single docker-compose command and run your first backtest in under 10 minutes.
Architecture Overview
Understand how the 16 adapters, MinIO bucket layout, and Redis minute-index work together to satisfy the backtest-kit write durability contract.
Adapters Overview
Full reference for every IPersist adapter — object key format, Redis index behaviour, upsert semantics, and cold-index fallback.