Persistence and Storage: From File to MongoDB and Redis

Backtest Kit’s persistence layer is fully pluggable. The framework defines 15 separate persistence contracts — one per domain (signals, candles, schedules, risk, partials, and more) — and ships default implementations that work out of the box. Swapping any or all of them for a production-grade backend requires a single setup() call and a few environment variables; your strategy code never changes.

Default: File-Based Storage

The default adapter writes every state mutation to disk using atomic file writes (writeFileAtomic). A partial write can never corrupt saved state because the OS swaps in the completed file only after the write succeeds.

./dump/signal/{strategyName}/{symbol}.json    ← open position state
./dump/candle/{exchangeName}/…               ← cached OHLCV data
./dump/backtest/{strategyName}.md            ← generated reports

Crash recovery flow:

Process crashes mid-trade.
On restart, ClientStrategy.waitForInit() reads the last persisted JSON from disk.
The engine resumes monitoring the open position from the exact state it was in before the crash.
No trades are lost; no duplicate signals are generated.

File-based storage requires zero configuration and is the right choice for single-symbol bots and development. It becomes a bottleneck under parallel multi-symbol backtests that perform thousands of context-keyed reads per second.

In-Memory Storage

For fast iteration during development or unit tests, swap individual adapters for in-memory implementations:

import { PersistSignalAdapter } from 'backtest-kit';

// Custom in-memory adapter — state lives in a plain Map
class MemorySignalAdapter {
  private store = new Map<string, unknown>();

  async readValue(entityId: string) {
    return this.store.get(entityId) ?? null;
  }

  async writeValue(entityId: string, entity: unknown) {
    this.store.set(entityId, entity);
  }
}

PersistSignalAdapter.usePersistSignalAdapter(MemorySignalAdapter);

In-memory adapters are not crash-safe and should only be used in backtest or test contexts where durability is not required.

Production: MongoDB + Redis

For production deployments, @backtest-kit/mongo replaces all 15 file-based adapters with MongoDB as the source of truth and Redis as an O(1) lookup cache. All 15 IPersist*Instance contracts are implemented — strategy code stays unchanged.

Installation

npm install @backtest-kit/mongo backtest-kit mongoose ioredis

Setup

Create config/setup.config.ts and call setup() before any trading operations:

import { setup } from '@backtest-kit/mongo';

// Reads all connection settings from environment variables.
// Call once at process startup — before addStrategySchema or Backtest.background.
setup();

To pass connection parameters explicitly (useful in Docker or CI):

import { setup } from '@backtest-kit/mongo';

setup({
  CC_MONGO_CONNECTION_STRING: 'mongodb://mongo:27017/backtest-kit',
  CC_REDIS_HOST: 'redis',
  CC_REDIS_PORT: 6379,
  CC_REDIS_PASSWORD: 'secret',
});

Environment Variables

CC_MONGO_CONNECTION_STRING=mongodb://localhost:27017/backtest-kit?wtimeoutMS=15000
CC_REDIS_HOST=127.0.0.1
CC_REDIS_PORT=6379
CC_REDIS_USER=default
CC_REDIS_PASSWORD=

Calling setup() disables all default file-based adapters and replaces them with MongoDB-backed implementations. After setup(), your process is the owner of the persistence layer — ensure MongoDB and Redis are reachable before calling Backtest.background() or Live.background().

How Redis O(1) Caching Works

Every domain has two layers: a DbService that talks to MongoDB and a CacheService that talks to Redis. When the engine reads state for a context key (for example symbol + strategyName + exchangeName for a signal), the DbService first asks Redis for the MongoDB _id. If the key is present, the document is fetched directly by _id — two O(1) operations total. On a cache miss, it falls back to an indexed MongoDB query and then writes the _id back to Redis so the next call is instant.

read signal for (BTCUSDT, my_strategy, binance)
  │
  ├─ Redis GET  → hit  → Mongo findById(_id)           ← O(1) + O(1)
  │
  └─ Redis GET  → miss → Mongo findOne(filter)
                       → Redis SET(_id)
                       → return document

After every write, the Redis entry is updated in the same round-trip, so a write followed immediately by a read always hits the cache. This is what enables ~6,300× aggregate replay speed in parallel multi-symbol backtests.

Atomic Writes

All writes use a single findOneAndUpdate round-trip with upsert: true:

const document = await SignalModel.findOneAndUpdate(
  { symbol, strategyName, exchangeName },   // unique compound index
  { $set: { payload } },
  { upsert: true, new: true, setDefaultsOnInsert: true },
);
await signalCacheService.setSignalId(readTransform(document.toJSON()));

MongoDB rejects any concurrent duplicate insert at the storage-engine level. The returned document is immediately written to Redis, making the next read O(1) with fresh data — no application-side locks, no retry loops under concurrent symbol writes.

Soft Delete

Measure, Interval, and Memory records are never physically deleted. Calling removeMeasureData, removeIntervalData, or removeMemoryData sets removed: true on the document. All listing operations automatically filter on removed: false. This preserves the audit trail for LLM reasoning history and backtest session data.

The 15 Persistence Contracts

Full list of IPersist*Instance contracts

Adapter	MongoDB Collection	Unique Index	Notes
Candle	`candle-items`	`symbol + interval + timestamp`	Immutable — first write wins
Signal	`signal-items`	`symbol + strategyName + exchangeName`	Core position state
Schedule	`schedule-items`	`symbol + strategyName + exchangeName`	Limit order scheduling
Risk	`risk-items`	`riskName + exchangeName`	Risk profile state
Partial	`partial-items`	`symbol + strategyName + exchangeName + signalId`	Partial profit/loss events
Breakeven	`breakeven-items`	`symbol + strategyName + exchangeName + signalId`	Breakeven SL moves
Storage	`storage-items`	`backtest + signalId`	General key-value storage per signal
Notification	`notification-items`	`backtest + notificationId`	Alert deduplication
Log	`log-items`	`entryId`	Structured log entries
Measure	`measure-items`	`bucket + entryKey`	LLM/API response cache (soft delete)
Interval	`interval-items`	`bucket + entryKey`	Time-window state (soft delete)
Memory	`memory-items`	`signalId + bucketName + memoryId`	Persistent LLM context (soft delete)
Recent	`recent-items`	`symbol + strategyName + exchangeName + frameName + backtest`	Last-seen signal per context
State	`state-items`	`signalId + bucketName`	Arbitrary strategy state per signal
Session	`session-items`	`strategyName + exchangeName + frameName`	Cached indicator calculations

API Reference

import { setup, install, setConfig, getConfig, getMongo, getRedis } from '@backtest-kit/mongo';

// Setup and register all 15 adapters (reads from env if no argument)
setup();

// Register adapters only (configuration already set via setConfig or env)
install();

// Override individual connection parameters at runtime
setConfig({ CC_REDIS_HOST: 'redis-prod', CC_REDIS_PORT: 6380 });

// Read the current merged configuration
const config = getConfig();

// Access the raw Mongoose instance (lazy singleton)
const mongoose = await getMongo();

// Access the raw ioredis instance (lazy singleton)
const redis = await getRedis();

Persistence Configuration per Mode

In production multi-symbol setups, you often want different adapters for backtest vs live runs. The config/setup.config.ts pattern from backtest-monorepo-parallel shows how:

import { setup } from '@backtest-kit/mongo';

// Live mode uses MongoDB for everything (durable)
// Backtest mode uses local file / memory for speed
if (process.env.MODE === 'live') {
  setup();
} else {
  // default file-based adapters remain active
}

Subsystem	Live	Backtest
Session	MongoDB (durable)	Local file
Storage	MongoDB	In-memory
Recent	MongoDB	In-memory
Notification	MongoDB	In-memory
Memory	MongoDB	Local file
State	MongoDB	Local file
Candle cache	MongoDB	MongoDB
Log	JSONL	JSONL

Get Started

Core Concepts

Guides

Strategy Examples

Persistence and Storage: From File to MongoDB and Redis

Default: File-Based Storage

In-Memory Storage

Production: MongoDB + Redis

Installation

Setup

Environment Variables

How Redis O(1) Caching Works

Atomic Writes

Soft Delete

The 15 Persistence Contracts

API Reference

Persistence Configuration per Mode

Build docs developers (and LLMs) love

Get Started

Core Concepts

Guides

Strategy Examples

Documentation Index

​Default: File-Based Storage

​In-Memory Storage

​Production: MongoDB + Redis

​Installation

​Setup

​Environment Variables

​How Redis O(1) Caching Works

​Atomic Writes

​Soft Delete

​The 15 Persistence Contracts

​API Reference

​Persistence Configuration per Mode

Build docs developers (and LLMs) love

Default: File-Based Storage

In-Memory Storage

Production: MongoDB + Redis

Installation

Setup

Environment Variables

How Redis O(1) Caching Works

Atomic Writes

Soft Delete

The 15 Persistence Contracts

API Reference

Persistence Configuration per Mode