Documentation Index
Fetch the complete documentation index at: https://mintlify.com/backtest-kit/backtest-kit-minio-s3-docker/llms.txt
Use this file to discover all available pages before exploring further.
MinIO is the primary persistence layer for every entity in backtest-kit-minio-s3-docker. Every JSON record — candles, signals, logs, notifications, and all 12 other entity types — is stored as an individual S3 object in MinIO. Because S3 provides strong read-after-write consistency for single objects, the backtest-kit write durability contract (readXData sees the value immediately after writeXData returns) is satisfied by plain object semantics, without transactions or secondary acknowledgements. Redis is strictly an index on top of this durable foundation; MinIO can stand alone even if Redis is empty.
Bucket Layout
All 16 entities share a single physical MinIO bucket named backtest-kit. Within that bucket, each entity type occupies its own root folder (prefix), so objects from different entities never collide in the same key namespace.
The BaseStorage factory class parses its constructor argument as "<bucket>/<root-folder>". The first path segment before the / becomes the physical S3 bucket name; everything after it becomes a transparent key prefix that is prepended to every object key operation:
// "backtest-kit/candle-items"
// └─ bucket: "backtest-kit"
// └─ rootPrefix: "candle-items/"
const [bucketName, ...folders] = BUCKET_NAME.split("/");
this.bucketName = bucketName;
this.rootPrefix = folders.length ? `${folders.join("/")}/` : "";
BaseStorage("breakeven-items")) means the entity gets a dedicated bucket of its own — the factory is fully backward compatible with that pattern. In this project, however, every service passes "backtest-kit/<entity>-items", keeping all data in one bucket and relying on prefix-scoped listings.
The full constructor from src/lib/common/BaseStorage.ts:
export const BaseStorage = factory(
class BaseStorage {
readonly loggerService = inject<LoggerService>(TYPES.loggerService);
readonly minioService = inject<MinioService>(TYPES.minioService);
/**
* Physical MinIO bucket: the first path segment of BUCKET_NAME.
* S3 bucket names cannot contain slashes, so "backtest-kit/candle-items"
* means bucket "backtest-kit" with root key prefix "candle-items/".
*/
readonly bucketName: string;
/** Root key prefix inside the bucket ("" when BUCKET_NAME has no parent folder). */
readonly rootPrefix: string;
constructor(public readonly BUCKET_NAME: string) {
const [bucketName, ...folders] = BUCKET_NAME.split("/");
this.bucketName = bucketName;
this.rootPrefix = folders.length ? `${folders.join("/")}/` : "";
}
// …
}
);
Object Key Schemas
Each entity family uses a key pattern that reflects how it will be queried:
Context-addressed entities (Signal, Schedule, Strategy, Risk, Partial, Breakeven, Recent, State, Session) use a key that is the exact tuple of their context fields:
signal-items/ symbol/strategyName/exchangeName
risk-items/ riskName/exchangeName
partial-items/ symbol/strategyName/exchangeName/signalId
PUT to the same key, with no risk of duplicates.
Candles add the candle timestamp as the final segment:
candle-items/ exchangeName/symbol/interval/timestamp
e.g. ccxt_binance/BTCUSDT/1h/1704067200000
const TIMESTAMP_PAD = String(Number.MAX_SAFE_INTEGER).length;
const GET_STORAGE_KEY_FN = (entryId: string, when: Date) => {
const inverted = String(Number.MAX_SAFE_INTEGER - when.getTime())
.padStart(TIMESTAMP_PAD, "0");
return `${inverted}_${entryId}`;
}
Number.MAX_SAFE_INTEGER − ms, zero-padded to a fixed width, produces keys that sort newest-first under plain lexicographic S3 LIST. This is the cold-index fallback that keeps listings correct even when Redis is empty.
Notification keys follow the same inverted-timestamp scheme, prefixed by backtest mode:
notification-items/ backtest/⟲ts_notificationId
LIST on their folders returns only live records:
measure-items/ bucket/entryKey
interval-items/ bucket/entryKey
memory-items/ signalId/bucket/memoryId
BaseStorage Operations
BaseStorage is a factory()-produced class that every data service extends. All object key arguments are relative to the entity’s rootPrefix — the prefix is applied transparently inside each method.
// Write a JSON value to key (creates or overwrites)
async set(key: string, value: unknown): Promise<void>
// Read a JSON value by key; returns null if not found
async get<T = unknown>(key: string | null): Promise<T | null>
// Cheaply test existence without downloading the object body (HEAD / statObject)
async has(key: string): Promise<boolean>
// Physically delete the object
async delete(key: string): Promise<void>
// Delete all objects under an optional prefix, in batches of 1 000
async clear(prefix?: string): Promise<void>
// Async-iterate object keys under an optional prefix, with optional limit
async *keys(prefix?: string, limit?: number): AsyncIterableIterator<string>
// Async-iterate object bodies (GET each key from keys())
async *values(prefix?: string, limit?: number): AsyncIterableIterator<unknown>
// Async-iterate [key, value] pairs
async *iterate(prefix?: string, limit?: number): AsyncIterableIterator<readonly [string, unknown]>
// Materialise all [key, value] pairs under a prefix into an array
async toArray(prefix?: string): Promise<[string, unknown][]>
// Count objects under a prefix (walks the full listing; use sparingly)
async size(prefix?: string): Promise<number>
set method serialises the value to UTF-8 JSON and calls putObject with Content-Type: application/json. The get method streams the response body into a Buffer and deserialises with JSON.parse. The has method calls statObject and returns false on NoSuchKey / NotFound — it never downloads the body, which makes it the right tool for the candle insert-only check.
Candle Insert-Only Pattern
Candles are immutable: the full record is completely determined by (symbol, interval, timestamp, open, high, low, close, volume). There is never a reason to overwrite a stored candle body. CandleDataService enforces this with a has check before any set:
// src/lib/services/data/CandleDataService.ts
const GET_STORAGE_KEY_FN = (symbol: string, interval: CandleInterval, timestamp: number) => {
return `${EXCHANGE_NAME}/${symbol}/${interval}/${timestamp}`;
}
public create = async (dto: ICandleDto): Promise<ICandleRow> => {
const key = GET_STORAGE_KEY_FN(dto.symbol, dto.interval, dto.timestamp);
const now = new Date();
const row: ICandleRow = {
id: key,
exchangeName: EXCHANGE_NAME,
symbol: dto.symbol,
interval: dto.interval,
timestamp: dto.timestamp,
open: dto.open,
high: dto.high,
low: dto.low,
close: dto.close,
volume: dto.volume,
createDate: now,
updatedDate: now,
};
// Candles are immutable: the row is fully determined by the dto, so an
// existence check (stat) replaces downloading the stored body.
if (await this.has(key)) {
return row;
}
await this.set(key, row);
return row;
};
has call is a cheap statObject (HEAD request). If the object already exists, the method returns immediately without a body download and without a PUT. A backtest replay of tens of thousands of candles therefore issues only one PUT per candle on the first run, and only statObject calls on every subsequent run — no bandwidth wasted re-uploading data that cannot have changed.
MinioService: Lazy Bucket Creation
MinioService wraps the MinIO client with a memoize-keyed getClient method. The first call for a given bucket name checks whether the bucket exists and creates it if not; subsequent calls for the same bucket return the cached client immediately:
// src/lib/services/base/MinioService.ts
import { memoize } from "functools-kit";
import { getMinio } from "../../../config/minio";
export class MinioService {
public getClient = memoize(
(bucketName) => bucketName,
async (bucketName: string) => {
const minioClient = getMinio();
if (await minioClient.bucketExists(bucketName)) {
return minioClient;
}
await minioClient.makeBucket(bucketName);
return minioClient;
}
);
}
bucketExists + optional makeBucket call for the lifetime of the process. Because this project consolidates everything into the single backtest-kit bucket, the creation check fires once at process start and is never repeated.
MinIO exposes S3-compatible bucket versioning and lifecycle policies via its web console at :9001. You can enable versioning on the backtest-kit bucket to keep a history of every object mutation, or add lifecycle rules to transition old candle objects to cheaper storage tiers. These are purely infrastructure-level features — no application code changes are needed.