Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/theonetrade/backtest-kit-redis-postgres-pgpool-docker/llms.txt

Use this file to discover all available pages before exploring further.

The persistence layer in this project is a three-tier stack that replaces backtest-kit’s default file-based ./dump/ storage with a production-grade infrastructure. PostgreSQL (via TypeORM) holds every record as the source of truth, Redis acts as an O(1) identity cache that maps context keys to row UUIDs, and Pgpool-II sits in front of the PostgreSQL cluster as a single :5432 entrypoint that routes writes to the primary node and load-balances reads across two streaming replicas. Strategy code, runners, and the backtest-kit CLI entrypoint are completely unchanged — only the persistence layer is swapped.

PostgreSQL

Source of truth for all 16 domain tables. TypeORM EntitySchema definitions, jsonb payloads, uuid primary keys, and timestamptz audit columns. Schema is created automatically on first connect via synchronize: true — no manual migration step needed.

Redis

O(1) ID cache. Stores only the row’s UUID mapped from its context key (e.g. exchangeName:strategyName:symbol). Seeded on every write from the RETURNING * result. Cache misses fall back to Postgres and backfill automatically.

Pgpool-II

Single :5432 entrypoint over a 1-primary + 2-replica streaming cluster. Writes go to the primary; reads are load-balanced across replicas. Exposes the replication lag that would hide read-after-write bugs on a single-node dev database.

​
The Three-Tier Stack

Every backtest-kit IPersist*Instance implementation follows a uniform skeleton. When the adapter is first touched, it gates on waitForInfra() — a singleshot barrier that waits for both PostgresService and RedisService to confirm their connections. After that, readXData delegates to ioc.xDbService.findByContext(...) and writeXData delegates to ioc.xDbService.upsert(...). 
PersistSignalAdapter.usePersistSignalAdapter(class implements IPersistSignalInstance {
  constructor(
    readonly symbol: string,
    readonly strategyName: string,
    readonly exchangeName: string,
  ) {}

  async waitForInit(initial: boolean) {
    if (!initial) return;
    await waitForInfra();        // gate first-touch on Postgres + Redis ready
  }

  async readSignalData(): Promise<ISignalRow | null> {
    const row = await ioc.signalDbService.findByContext(
      this.symbol, this.strategyName, this.exchangeName,
    );
    return row ? row.payload : null;
  }

  async writeSignalData(signalRow: ISignalRow | null): Promise<void> {
    await ioc.signalDbService.upsert(
      this.symbol, this.strategyName, this.exchangeName, signalRow,
    );
  }
});

​
The 16 Persist Adapters

Each of the 16 adapters maps to a PostgreSQL table with a unique compound index whose columns form the context key. The DI container (ioc) wires the full graph — base services, cache services, and db services — lazily via di-factory and inject.
AdapterTableContext Key (unique index)
Candlecandle-items(symbol, interval, timestamp)
Signalsignal-items(symbol, strategyName, exchangeName)
Scheduleschedule-items(symbol, strategyName, exchangeName)
Strategystrategy-items(symbol, strategyName, exchangeName)
Riskrisk-items(riskName, exchangeName)
Partialpartial-items(symbol, strategyName, exchangeName, signalId)
Breakevenbreakeven-items(symbol, strategyName, exchangeName, signalId)
Storagestorage-items(backtest, signalId)
Notificationnotification-items(backtest, notificationId)
Loglog-items(entryId)
Measuremeasure-items(bucket, entryKey)
Intervalinterval-items(bucket, entryKey)
Memorymemory-items(signalId, bucketName, memoryId)
Recentrecent-items(symbol, strategyName, exchangeName, frameName, backtest)
Statestate-items(signalId, bucketName)
Sessionsession-items(strategyName, exchangeName, frameName, symbol, backtest)

​
DI Container Wiring

The ioc object is the root of the dependency-injection graph. It exposes three layers of services, all constructed lazily on first access:
  • LoggerService — structured logger injected into every other service for loggerService.log(...) calls.
  • PostgresService — calls getPostgres() (a singleshot DataSource initializer) and exposes waitForInit() so adapters can block until the schema is ready.
  • RedisService — calls getRedis() (a singleshot ioredis client) and exposes waitForInit().
One per domain (e.g. SignalCacheService, CandleCacheService). Each extends BaseMap with a namespaced Redis prefix and a typed key-builder method (e.g. _cacheKey(symbol, strategyName, exchangeName)). TTL is -1 (no expiry) for identity caches.
One per domain (e.g. SignalDbService, IntervalDbService). Each extends BaseCRUD with domain-specific upsert, findByContext / findByKey, and (for soft-delete adapters) softRemove and listKeys methods.

​
BaseCRUD and BaseMap Abstractions

Both abstractions are factory-generated classes that accept constructor parameters and compose via inject. BaseCRUD(TargetModel) wraps a TypeORM EntitySchema and provides create, update, findById, findByFilter, and findAll methods. It resolves the DataSource through the singleton getPostgres() promise on every call, so it is safe to call before the database is connected — the promise simply resolves once and is reused. Domain DbServices inherit BaseCRUD and add the atomic upsert query builder on top. BaseMap(connectionKey, ttlExpireSeconds) wraps the singleton ioredis client and provides a full string-keyed map API: get, set, delete, has, clear, toArray, iterate, keys, values, and size. All keys are namespaced as ${connectionKey}:${key}, and SCAN-based iteration is used for toArray, keys, values, and size to avoid blocking the Redis event loop on large key sets. Domain CacheServices inherit BaseMap and add typed key-builder methods on top.

​
Data Flow

​
Write Path

1

Adapter calls upsert on DbService

The adapter (e.g. writeSignalData) passes the context fields and the domain payload to signalDbService.upsert(symbol, strategyName, exchangeName, payload).
2

Single atomic INSERT … ON CONFLICT … RETURNING *

The DbService executes one INSERT … ON CONFLICT (unique index) DO UPDATE SET payload = EXCLUDED.payload RETURNING * against the primary PostgreSQL node. No separate read happens. The written row is returned in the same statement.
3

Cache seeded from RETURNING row

The id field from the returned row is immediately written to Redis via cacheService.setXId(result). The cache key maps the context tuple to the UUID. Because this comes from the primary (not a replica), there is no replication-lag risk.

​
Read Path

1

Cache lookup (O(1) steady state)

findByContext calls cacheService.getXId(...). This is a single Redis GET returning the row’s UUID if the cache is warm.
2

PK lookup if cache hit

If a UUID was returned, super.findByFilter({ id: cachedId }) fetches the full row by primary key — also O(1) on the indexed uuid column. The result is returned immediately.
3

Postgres filter + cache backfill on miss

If the cache is cold (Redis restart, first access, eviction), findByFilter({ symbol, strategyName, exchangeName }) runs a B-tree lookup on the unique compound index. The result is then written back to Redis via cacheService.setXId(result) so the next call is a cache hit.
A Redis restart causes a cold-start on the next access for every context key, but no data is lost. The Postgres compound-index filter re-populates the cache entry on the first miss. Hot steady-state performance is restored within one request per context key.

​
Pgpool-II Cluster

The docker/pgpool/docker-compose.yaml boots the published tripolskypetr/pgpool:latest all-in-one image: one primary PostgreSQL node plus two streaming replicas, all behind Pgpool-II on port 5432. The application connects to that single port and never needs to know the primary’s address directly. 
services:
  pgpool:
    image: tripolskypetr/pgpool:latest
    container_name: pgcluster
    ports:
      - "5432:5432"   # single entrypoint: writes → primary, reads → replicas
    environment:
      POSTGRES_USER: backtest
      POSTGRES_PASSWORD: mysecurepassword
      POSTGRES_DB: backtest-pro
      REPL_USER: replicator
      REPL_PASSWORD: replicatorpass
The first boot takes approximately 60–90 seconds while the two replicas clone from the primary. The healthcheck (start_period: 120s, retries: 15) keeps the container in a starting state until Pgpool-II is accepting connections. Do not attempt to connect before the healthcheck passes.
The dev cluster is intentional: it exposes the replication lag that would silently hide read-after-write bugs behind the global-mutex illusion of a single PostgreSQL process. Any code that relies on a follow-up SELECT after a write will fail here — which is exactly the class of bug this architecture eliminates. See the Atomicity page for a full explanation.

Build docs developers (and LLMs) love

Get started for freeTalk to us