Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/theonetrade/backtest-kit-redis-postgres-pgpool-docker/llms.txt

Use this file to discover all available pages before exploring further.

backtest-kit-redis-postgres-pgpool-docker is a production-grade persistence layer for the backtest-kit algorithmic trading framework. This page explains what the project does, how its dual-store architecture works, and where each piece fits so you can confidently deploy or extend it.

​
What Problem It Solves

Out of the box, backtest-kit persists all strategy state to a local ./dump/ directory using JSON files. That works perfectly for a single developer running a backtest on their laptop, but it hits a wall the moment you need to:
  • Run in a container where the filesystem is ephemeral
  • Scale horizontally with multiple runners sharing state
  • Audit or query historical signal state with SQL
  • Survive process restarts in a long-running live-trading deployment
This project swaps the file-based layer for PostgreSQL (durable source of truth) and Redis (O(1) lookup cache) while leaving strategy code, runners, and the CLI entrypoint completely untouched.
Strategy logic, frame definitions, and exchange adapter registrations are unchanged. Only the persistence adapters are replaced — your trading logic does not need to know about PostgreSQL or Redis.

​
The 16 Persist Adapters

backtest-kit defines a family of IPersist*Instance contracts — one per domain object. This project ships 16 custom adapters that implement every contract on top of PostgreSQL (via TypeORM) and Redis, then register them all in src/config/setup.ts. Every adapter follows the same skeleton: 
PersistXAdapter.usePersistXAdapter(class implements IPersistXInstance {
  constructor(/* context fields injected by backtest-kit */) {}

  async waitForInit(initial: boolean) {
    if (!initial) return;
    await waitForInfra(); // gate first-touch on Postgres + Redis being ready
  }

  async readXData(...) {
    return await ioc.xDbService.findByContext(...);
  }

  async writeXData(..., when: Date) {
    await ioc.xDbService.upsert(..., when);
  }
});
AdapterTableUnique index (context key)Purpose
Candlecandle-items(symbol, interval, timestamp)OHLCV cache; immutable inserts
Signalsignal-items(symbol, strategyName, exchangeName)Live signal state per context
Scheduleschedule-items(symbol, strategyName, exchangeName)Pending scheduled signal
Strategystrategy-items(symbol, strategyName, exchangeName)Persistent strategy state per context
Riskrisk-items(riskName, exchangeName)Active risk positions snapshot
Partialpartial-items(symbol, strategyName, exchangeName, signalId)Partial profit/loss levels per signal
Breakevenbreakeven-items(symbol, strategyName, exchangeName, signalId)Breakeven reached flag
Storagestorage-items(backtest, signalId)Closed/opened signal log per mode
Notificationnotification-items(backtest, notificationId)Event notifications
Loglog-items(entryId)Strategy log entries
Measuremeasure-items(bucket, entryKey)LLM/API response cache (soft-delete)
Intervalinterval-items(bucket, entryKey)Once-per-interval markers (soft-delete)
Memorymemory-items(signalId, bucketName, memoryId)Per-signal memory store (soft-delete)
Recentrecent-items(symbol, strategyName, exchangeName, frameName, backtest)Last public signal per context
Statestate-items(signalId, bucketName)Per-signal state buckets
Sessionsession-items(strategyName, exchangeName, frameName, symbol, backtest)One session per running strategy
All tables and unique indexes are created automatically on first connect via TypeORM synchronize: true. There is no manual migration step.

​
Dual-Store Architecture

The project maintains two stores that work in concert: PostgreSQL is the durable source of truth, and Redis is an O(1) ID cache that accelerates the hot read path.

PostgreSQL (TypeORM)

Every write goes through a single INSERT … ON CONFLICT … DO UPDATE … RETURNING * statement. There is no read-then-write race — the row is either inserted or updated atomically, and the just-written row is returned in the same round-trip. UUID primary keys and createDate/updateDate columns are managed by TypeORM automatically.

Redis (ioredis)

After each upsert, the returned row’s id is stored in Redis under a composite cache key (e.g. exchangeName:strategyName:symbol). The next readXData call resolves the id with a single Redis GET, then fetches by primary key — O(1) instead of an indexed B-tree scan. On cache miss, Postgres is queried and Redis is backfilled.

​
Why atomicity matters

backtest-kit has a hard read-after-write durability contract: after writeXData() returns, the very next readXData() must see the written value. A naïve SELECT → INSERT or UPDATE pattern breaks this under concurrent access because two parallel writers can both see “no existing row” and both attempt an insert, causing a unique-constraint violation. The atomic INSERT … ON CONFLICT … DO UPDATE … RETURNING * pattern eliminates this entirely. PostgreSQL serializes concurrent inserts on the unique index; the losing writer takes the DO UPDATE branch instead of throwing. The RETURNING * clause seeds the Redis cache from the same statement — no follow-up SELECT that could hit a lagging replica.

​
Look-ahead bias protection

Adapters that influence signal logic (Risk, Partial, Breakeven, Recent, State, Session, Memory, Interval) store the when: Date simulation timestamp as a bigint epoch column. The when value is written in the same atomic upsert, letting backtest-kit’s internal filter verify that no read returns a value written in the future of the current simulation time.

​
The Pgpool-II Cluster

The development infrastructure in docker/pgpool/ runs a real cluster: one primary PostgreSQL node, two streaming replicas, and a Pgpool-II proxy — all in a single container exposed on :5432. Writes are routed to the primary; reads are load-balanced across the replicas.
First boot takes 60–90 seconds while the two replicas clone from the primary. The container’s healthcheck has a 120-second start_period to account for this. Do not run your application until the container is reported healthy.
This cluster topology exists on purpose. A single-node Postgres instance hides stale-read bugs behind an internal “global mutex” illusion — all reads hit the same process that just committed, so a write followed by a separate SELECT always looks correct. Add async replicas and that illusion breaks: a SELECT can be routed to a node that hasn’t replicated the commit yet. Running a real replica cluster in development surfaces these bugs before they reach production. For CI pipelines and simple local testing, a plain single-node Postgres is available at docker/postgres/docker-compose.yaml.

​
Three Running Modes

The mode-specific entry points in src/main/ all gate on getArgs() flags before connecting to the infrastructure:

Backtest

Replays historical candles over a fixed time window. Passes --backtest on the CLI or MODE=backtest in Docker. Calls warmCandles() to pre-fetch OHLCV data, then runs Backtest.background().

Live

Connects to a live exchange feed. Passes --live on the CLI or MODE=live in Docker. Runs Live.background() without candle warming — no look-ahead bias window needed.

Paper

Same runtime path as live trading but without real order execution. Passes --paper on the CLI or MODE=paper in Docker. Useful for forward-testing a strategy before going live.
Every mode initializes Postgres and Redis before doing anything else: 
await ioc.postgresService.waitForInit();
await ioc.redisService.waitForInit();

​
Feature Overview

Atomic ON CONFLICT Upserts

Every write is a single INSERT … ON CONFLICT … DO UPDATE … RETURNING * round-trip. No unique-constraint violations, no silent state corruption under concurrent access.

O(1) Redis Cache

*CacheService classes store row IDs in Redis under composite keys. Steady-state reads are one Redis GET + one Postgres PK lookup, regardless of table size.

Replica Cluster in Dev

docker/pgpool runs a 1-primary + 2-replica Pgpool-II cluster so stale-read bugs surface in development, not in production.

Zero Strategy Changes

All 16 adapters implement the standard IPersist*Instance contracts. Your strategy code, frames, and exchange modules require no modifications.

Auto Schema Sync

TypeORM synchronize: true creates every table and unique index on first connect. No migration files to manage.

Look-Ahead Bias Guard

Signal-affecting adapters store the simulation when timestamp so backtest-kit’s internal filter can reject future-dated reads during replay.

Soft-Delete Support

Measure, Interval, and Memory use server-side jsonb_set soft deletes — no SELECT-then-save window, no data loss.

One-Command Docker Deploy

docker-compose up -d with MODE, ENTRY, and UI env vars starts the full backtest-kit container alongside your PostgreSQL and Redis infrastructure.

​
Relationship to backtest-kit

This project is a first-party persistence integration for backtest-kit v15. It consumes the backtest-kit, @backtest-kit/cli, @backtest-kit/ui, @backtest-kit/graph, @backtest-kit/signals, and @backtest-kit/pinets packages at version 15.2.0 and is wired into the CLI’s standard entry-point convention (--entry flag guards every main() function). The persistence layer itself lives entirely in src/lib/ and src/config/setup.ts. The src/logic/ and src/main/ directories hold the example strategy and exchange adapter — they are the consumer of the persistence layer, not part of it.
Ready to run your first backtest? Head to the Quickstart to spin up the cluster and execute in under five minutes.

Build docs developers (and LLMs) love

Get started for freeTalk to us