MinIO S3 object storage layout and BaseStorage API reference

All persistent data in this project is stored as JSON objects in a single MinIO bucket named backtest-kit. There are no tables, no schemas, and no migrations — each entity type occupies a logical folder under that bucket, and every record is a self-contained JSON file addressed by a deterministic key. MinioService lazily creates the bucket the first time an adapter touches it, so there is no manual setup step.

Bucket Layout

The physical layout inside the backtest-kit bucket mirrors the adapter table:

backtest-kit/
├── candle-items/
│   └── ccxt_binance/BTCUSDT/1h/1700000000000
├── signal-items/
│   └── BTCUSDT/Jan2026Strategy/ccxt_binance
├── log-items/
│   └── 9007199254639991_abc123
├── notification-items/
│   └── false/9007199254639991_notif456
├── storage-items/
│   └── true/signal-789
└── ...

Each folder is a root key prefix; there are no actual S3 “directories” — MinIO simply lists objects whose names start with a given prefix.

BaseStorage: Constructor and Prefix Parsing

BaseStorage is a factory-generated class that maps a human-readable name like "backtest-kit/candle-items" to a physical bucket name and a transparent key prefix:

BaseStorage.ts

constructor(public readonly BUCKET_NAME: string) {
  // "backtest-kit/candle-items" → bucket="backtest-kit", rootPrefix="candle-items/"
  // "breakeven-items"           → bucket="breakeven-items",  rootPrefix=""
  const [bucketName, ...folders] = BUCKET_NAME.split("/");
  this.bucketName = bucketName;
  this.rootPrefix = folders.length ? `${folders.join("/")}/` : "";
}

All subsequent operations on BaseStorage prepend rootPrefix to every key before calling the MinIO client, so callers always work with short entity-relative keys:

// CandleDataService extends BaseStorage("backtest-kit/candle-items")
await this.set("ccxt_binance/BTCUSDT/1h/1700000000000", row);
// → PUT backtest-kit/candle-items/ccxt_binance/BTCUSDT/1h/1700000000000

BaseStorage Public API

Every data service inherits the following methods from BaseStorage. All network I/O goes through the MinIO Node.js SDK (minio package).

`set(key, value): Promise<void>`

Serialises value to UTF-8 JSON and writes it as a single MinIO object. The Content-Type header is set to application/json.

BaseStorage.ts

async set(key: string, value: unknown): Promise<void> {
  if (!key) throw new Error("Key cannot be empty");
  const minioClient = await this.minioService.getClient(this.bucketName);
  const buffer = Buffer.from(JSON.stringify(value), "utf-8");
  await minioClient.putObject(
    this.bucketName,
    this.rootPrefix + key,
    buffer,
    buffer.length,
    { "Content-Type": "application/json" }
  );
}

`get<T>(key): Promise<T | null>`

Downloads and JSON-parses the object at key. Returns null for NoSuchKey / NotFound errors; re-throws all other errors. Stream chunks are collected via createAwaiter from functools-kit — the method awaits the stream’s end event before parsing.

BaseStorage.ts

async get<T = unknown>(key: string | null): Promise<T | null> {
  if (key === null) return null;
  const minioClient = await this.minioService.getClient(this.bucketName);
  let dataStream: Readable;
  try {
    dataStream = await minioClient.getObject(this.bucketName, this.rootPrefix + key);
  } catch (error) {
    if (isNotFound(error)) return null;
    throw error;
  }
  const [awaiter, { resolve, reject }] = createAwaiter<Buffer>();
  const chunks: Uint8Array[] = [];
  dataStream.on("data", (chunk: Uint8Array) => { chunks.push(chunk); });
  dataStream.on("end", () => { resolve(Buffer.concat(chunks)); });
  dataStream.on("error", (error: Error) => { reject(error); });
  const buffer = await awaiter;
  return JSON.parse(buffer.toString("utf-8")) as T;
}

`has(key): Promise<boolean>`

Issues a statObject (HEAD) call — downloads no body. Returns false for not-found errors.

BaseStorage.ts

async has(key: string): Promise<boolean> {
  if (key === null) return false;
  const minioClient = await this.minioService.getClient(this.bucketName);
  try {
    await minioClient.statObject(this.bucketName, this.rootPrefix + key);
    return true;
  } catch (error) {
    if (isNotFound(error)) return false;
    throw error;
  }
}

`delete(key): Promise<void>`

Physically removes the object from MinIO. Used by soft-delete entities (Measure, Interval, Memory) to keep listKeys clean.

BaseStorage.ts

async delete(key: string): Promise<void> {
  if (key === null) return;
  const minioClient = await this.minioService.getClient(this.bucketName);
  await minioClient.removeObject(this.bucketName, this.rootPrefix + key);
}

`clear(prefix?): Promise<void>`

Deletes all objects whose keys start with prefix (default: all objects under rootPrefix). Batches deletions in groups of 1 000 so memory stays O(batch), not O(bucket).

BaseStorage.ts

async clear(prefix = ""): Promise<void> {
  const minioClient = await this.minioService.getClient(this.bucketName);
  let batch: string[] = [];
  for await (const key of this.keys(prefix)) {
    batch.push(this.rootPrefix + key);
    if (batch.length >= 1_000) {
      await minioClient.removeObjects(this.bucketName, batch);
      batch = [];
    }
  }
  if (batch.length) {
    await minioClient.removeObjects(this.bucketName, batch);
  }
}

`keys(prefix?, limit?): AsyncIterableIterator<string>`

Streams object names from MinIO’s listObjectsV2 (recursive), stripping the rootPrefix so callers receive entity-relative keys. Stops early when limit is reached.

BaseStorage.ts

async *keys(prefix = "", limit?: number): AsyncIterableIterator<string> {
  const minioClient = await this.minioService.getClient(this.bucketName);
  const objectStream = minioClient.listObjectsV2(
    this.bucketName,
    this.rootPrefix + prefix,
    true  // recursive
  );
  let count = 0;
  for await (const item of objectStream) {
    if (!item.name) continue;
    yield item.name.slice(this.rootPrefix.length);
    if (limit !== undefined && ++count >= limit) return;
  }
}

`values(prefix?, limit?): AsyncIterableIterator<unknown>`

Yields the parsed JSON body of each object returned by keys. Skips objects whose body is null (deleted between list and get).

`iterate(prefix?, limit?): AsyncIterableIterator<readonly [string, unknown]>`

Yields [key, value] tuples — useful for bulk migration or inspection.

`toArray(prefix?): Promise<[string, unknown][]>`

Eagerly materialises iterate into an array. Use with a scoped prefix to avoid pulling an entire bucket into memory.

`size(prefix?): Promise<number>`

Counts objects by walking keys — no body downloads.

MinioService: Lazy Bucket Creation

MinioService.getClient(bucketName) is a memoised async function: the first call for a given bucket name checks whether it exists and creates it if not; subsequent calls return the same client immediately without any network round trip.

MinioService.ts

export class MinioService {
  public getClient = memoize(
    (bucketName) => bucketName,   // cache key
    async (bucketName: string) => {
      const minioClient = getMinio();
      if (await minioClient.bucketExists(bucketName)) {
        return minioClient;
      }
      await minioClient.makeBucket(bucketName);
      return minioClient;
    }
  );
}

This means no pre-flight bucket provisioning is required in your Docker Compose setup. On first run, backtest-kit is created automatically. Subsequent process restarts skip the makeBucket call entirely because the bucket already exists — the bucketExists check handles the idempotency.

Candle Object Key Format

CandleDataService (which extends BaseStorage("backtest-kit/candle-items")) constructs its object keys from four fields:

CandleDataService.ts

const EXCHANGE_NAME = "ccxt_binance";

const GET_STORAGE_KEY_FN = (
  symbol: string,
  interval: CandleInterval,
  timestamp: number
) => {
  return `${EXCHANGE_NAME}/${symbol}/${interval}/${timestamp}`;
};

A concrete example:

candle-items/ccxt_binance/BTCUSDT/1h/1700000000000

This key is a pure function of the candle’s content: given the same (symbol, interval, timestamp) triple you always get the same key, with no UUID generation or sequence counters.

Candle Insert-Only Semantics

Candles are immutable market data. CandleDataService.create never overwrites an existing object — it uses has() (a zero-body statObject) to check existence before calling set():

CandleDataService.ts

public create = async (dto: ICandleDto): Promise<ICandleRow> => {
  const key = GET_STORAGE_KEY_FN(dto.symbol, dto.interval, dto.timestamp);
  const now = new Date();
  const row: ICandleRow = {
    id: key,
    exchangeName: EXCHANGE_NAME,
    symbol: dto.symbol,
    interval: dto.interval,
    timestamp: dto.timestamp,
    open: dto.open,
    high: dto.high,
    low: dto.low,
    close: dto.close,
    volume: dto.volume,
    createDate: now,
    updatedDate: now,
  };
  // Candles are immutable: the row is fully determined by the dto,
  // so a stat (HEAD) replaces downloading the stored body.
  if (await this.has(key)) {
    return row;
  }
  await this.set(key, row);
  return row;
};

This achieves two goals simultaneously:

No re-downloads — a warm candle cache costs only 1 HEAD per candle, not 1 GET.
No overwrites — if two strategy instances race to write the same candle, both produce identical data, so whichever PUT lands first is the canonical copy.

The MinIO web console is available at port 9001 (configured in docker/minio/docker-compose.yaml). You can browse buckets, inspect object metadata, and manually purge entries from the console UI during development. The S3 API itself is served on port 9000.

Get Started

Architecture

Configuration

Persist Adapters

Strategy Development

MinIO S3 object storage layout and BaseStorage API reference

Bucket Layout

BaseStorage: Constructor and Prefix Parsing

BaseStorage Public API

`set(key, value): Promise<void>`

`get<T>(key): Promise<T | null>`

`has(key): Promise<boolean>`

`delete(key): Promise<void>`

`clear(prefix?): Promise<void>`

`keys(prefix?, limit?): AsyncIterableIterator<string>`

`values(prefix?, limit?): AsyncIterableIterator<unknown>`

`iterate(prefix?, limit?): AsyncIterableIterator<readonly [string, unknown]>`

`toArray(prefix?): Promise<[string, unknown][]>`

`size(prefix?): Promise<number>`

MinioService: Lazy Bucket Creation

Candle Object Key Format

Candle Insert-Only Semantics

Build docs developers (and LLMs) love

Get Started

Architecture

Configuration

Persist Adapters

Strategy Development

Documentation Index

​Bucket Layout

​BaseStorage: Constructor and Prefix Parsing

​BaseStorage Public API

​set(key, value): Promise<void>

​get<T>(key): Promise<T | null>

​has(key): Promise<boolean>

​delete(key): Promise<void>

​clear(prefix?): Promise<void>

​keys(prefix?, limit?): AsyncIterableIterator<string>

​values(prefix?, limit?): AsyncIterableIterator<unknown>

​iterate(prefix?, limit?): AsyncIterableIterator<readonly [string, unknown]>

​toArray(prefix?): Promise<[string, unknown][]>

​size(prefix?): Promise<number>

​MinioService: Lazy Bucket Creation

​Candle Object Key Format

​Candle Insert-Only Semantics

Build docs developers (and LLMs) love

Bucket Layout

BaseStorage: Constructor and Prefix Parsing

BaseStorage Public API

`set(key, value): Promise<void>`

`get<T>(key): Promise<T | null>`

`has(key): Promise<boolean>`

`delete(key): Promise<void>`

`clear(prefix?): Promise<void>`

`keys(prefix?, limit?): AsyncIterableIterator<string>`

`values(prefix?, limit?): AsyncIterableIterator<unknown>`

`iterate(prefix?, limit?): AsyncIterableIterator<readonly [string, unknown]>`

`toArray(prefix?): Promise<[string, unknown][]>`

`size(prefix?): Promise<number>`

MinioService: Lazy Bucket Creation

Candle Object Key Format

Candle Insert-Only Semantics