Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/theonetrade/backtest-kit-minio-s3-docker/llms.txt

Use this file to discover all available pages before exploring further.

backtest-kit enforces a write durability contract: after writeXData(...) returns, the very next readXData(...) call from any code path must see the value that was just written. In a traditional database this contract is upheld by transactions or write-ahead logs. This project meets the same guarantee using plain S3 object semantics — no transactions, no two-phase commits, no sequence numbers.
Amazon S3 (and MinIO) provides strong read-after-write consistency for new objects as of late 2020. A PUT that completes successfully is immediately visible to any subsequent GET or HEAD for that exact key. This project relies on this guarantee as its foundation.

​
Four Durability Principles

1

Deterministic keys — upserts are single idempotent PUTs

Every record’s object key is a pure function of its context fields. There is no UUID generation at write time, no sequence counter, and no read-before-write step to discover an existing key.
AdapterKey function
Candleccxt_binance/{symbol}/{interval}/{timestamp}
Signal{symbol}/{strategy}/{exchange}
Risk{riskName}/{exchange}
Partial{symbol}/{strategy}/{exchange}/{signalId}
Session{strategy}/{exchange}/{frame}/{symbol}/{backtest}
Because the key is deterministic, an upsert is a single PUT. Two concurrent writers for the same context produce the same key and the same serialised value, so the last-write-wins S3 semantic is also correct-by-construction — there is no “wrong” winner.
2

Immutable entities — candles use stat+PUT, logs skip already-persisted keys

Two categories of entities are never rewritten after their first creation:Candles are market data: the OHLCV values for (exchange, symbol, interval, timestamp) never change. CandleDataService.create issues a statObject (HEAD) before writing. If the object exists, the method returns immediately without downloading the body:
CandleDataService.ts
public create = async (dto: ICandleDto): Promise<ICandleRow> => {
  const key = GET_STORAGE_KEY_FN(dto.symbol, dto.interval, dto.timestamp);
  const now = new Date();
  const row: ICandleRow = {
    id: key,
    exchangeName: EXCHANGE_NAME,
    symbol: dto.symbol,
    interval: dto.interval,
    timestamp: dto.timestamp,
    open: dto.open,
    high: dto.high,
    low: dto.low,
    close: dto.close,
    volume: dto.volume,
    createDate: now,
    updatedDate: now,
  };
  // Candles are immutable: the row is fully determined by the dto,
  // so a stat (HEAD) replaces downloading the stored body.
  if (await this.has(key)) {
    return row;
  }
  await this.set(key, row);
  return row;
};
Log and Notification entries are also immutable once written. LogDataService.upsert checks an in-process Set<string> of already-persisted keys first (FIFO-capped at 10 000 entries), then falls back to a statObject for keys not yet seen in this process. If either check confirms the object exists, the PUT and Redis register are both skipped entirely:
LogDataService.ts
public upsert = async (entryId: string, payload: ILogEntry): Promise<void> => {
  const key = GET_STORAGE_KEY_FN(entryId, new Date(payload.priority));
  // Fast in-process check first — zero network cost for already-seen keys
  if (this._persistedKeys.has(key)) {
    return;
  }
  // Fall back to a stat for keys not in the in-process index
  if (await this.has(key)) {
    this._rememberKey(key);
    return;
  }
  const now = new Date();
  const row: ILogRow = { id: key, entryId, payload, createDate: now, updatedDate: now };
  // MinIO first, Redis second — see Principle 3
  await this.set(key, row);
  await this.logConnectionService.register(key);
  this._rememberKey(key);
};
The in-process index is important because backtest-kit re-sends the full accumulated entry list on every writeLogData call. Without the in-process index, a list of 200 entries would cost 200 statObject calls on every tick; with it, the cost is zero for all entries that have been seen in the current process lifetime.
3

Write order: MinIO first, Redis second

For entities that use the Redis time index (Log, Notification, Storage), the write sequence is always:
  1. this.set(key, row) → MinIO PUT
  2. this.logConnectionService.register(key) → Redis SADD + SETNX
This ordering ensures that the Redis index never contains a pointer to a non-existent MinIO object. A crash or process kill between step 1 and step 2 leaves the object readable by its deterministic key, but absent from listNewest until the Redis index is rebuilt (either by a later register call or by the cold-index fallback path).The reverse — writing Redis first — would create a phantom entry: listNewest would return the key, get(key) would return null, and the caller would silently drop a log entry it believed was persisted.
4

`removed` means absent — physical deletes keep listKeys clean

Soft-delete entities (Measure, Interval, Memory) implement removal by physically deleting the MinIO object rather than writing a tombstone:
src/config/setup.ts
async removeMeasureData(key: string): Promise<void> {
  await ioc.measureDataService.softRemove(this.bucket, key);
}
softRemove calls BaseStorage.delete(key), which issues removeObject to MinIO. After this call:
  • readMeasureData(key) calls get(key), which returns null for NoSuchKey — no special null-check logic is needed.
  • listMeasureData() calls BaseStorage.keys(prefix) — a pure listObjectsV2 stream. Deleted objects are never returned. The listing is O(surviving objects), not O(total historical objects).
This is in contrast to a tombstone approach (writing { removed: true }) where every listing would need to download and filter objects, and the bucket would grow without bound over time.

​
Crash Safety Summary

Crash pointState in MinIOState in RedisEffect on reads
Before set(key, row)Object absentNot registeredreadXData returns previous value (or null) — correct
After set, before registerObject presentNot registeredPoint read by key returns the value ✓; does not appear in listNewest until cold fallback or next register
After both set and registerObject presentRegisteredFully visible to all read paths ✓
After delete (soft-remove)Object absentN/AreadXData returns null ✓; absent from listKeys ✓
There is a narrow crash window between the MinIO PUT completing and the Redis SADD being sent for Log, Notification, and Storage entries. During this window the object is readable by its deterministic key (e.g. via findByEntryId) but invisible to listNewest. This is not a data loss scenario — the object exists in MinIO. The next time listAll runs and the Redis floor key is absent, the cold-index fallback will discover the object through listObjectsV2 and re-register it, fully restoring visibility. The window is typically sub-millisecond under normal operating conditions.

​
Adapter waitForInit Gate

Every adapter delays its first operation until the infrastructure is confirmed ready:
src/config/setup.ts
const waitForInfra = singleshot(async () => {
  await ioc.redisService.waitForInit();
  // MinIO client connects lazily per bucket — no explicit wait needed
});

// Inside each adapter:
async waitForInit(initial: boolean) {
  if (!initial) return;
  await waitForInfra();
}
RedisService.waitForInit waits up to 15 seconds for the Redis ready event. The MinIO client is created lazily by MinioService.getClient on first bucket access, so there is no up-front TCP handshake to gate on. If Redis does not become ready within the timeout, the error propagates and the process exits — the strategy never starts writing to a partially initialised store.

Build docs developers (and LLMs) love

Get started for freeTalk to us