Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/NikolayS/PgQue/llms.txt

Use this file to discover all available pages before exploring further.

PgQue is a two-layer system built on top of PgQ’s proven core engine. Understanding how ticks, batches, and rotation work explains both the zero-bloat guarantee and the transaction boundary requirements.

​
Two-layer architecture

pgque-core is a mechanical repackaging of PgQ: renamed for the pgque schema, modernized for PostgreSQL 14+ (pg_snapshot, xid8), hardened with security definers and role grants, and bundled into a single SQL file. The ~4,000 lines of PL/pgSQL are inherited from PgQ’s decade-plus production history at Skype. pgque-api is the modern convenience layer (~1,500 lines): send, receive, ack, nack, the dead letter queue, delayed delivery, and the cooperative consumer API. Every API function reduces cleanly to pgque-core primitives — send() calls insert_event(), receive() calls next_batch() + get_batch_events(), ack() calls finish_batch().

​
Snapshot-based batching

Unlike SKIP LOCKED queues that operate row by row, PgQue creates ticks — snapshot markers in the event stream. A tick records the PostgreSQL snapshot at the moment it runs. A batch is the set of events between two consecutive ticks, as seen through those snapshots. This design has two key consequences:
  1. Zero dead tuples in the hot path. There are no UPDATE or DELETE operations on event rows. Events accumulate in the current rotation table until the table is TRUNCATEd and swapped out.
  2. The snapshot rule. send → ticker → receive must each run in separate committed transactions. The ticker’s snapshot must be taken after send commits; receive only returns events visible in the snapshot recorded by the most recent tick.

​
The snapshot rule

Transaction 1: pgque.send(...)         -- insert event row, commit
Transaction 2: pgque.ticker()          -- record snapshot, commit
Transaction 3: pgque.receive(...)      -- read events visible in that snapshot
Combining any of these in one BEGIN/COMMIT block produces empty batches: 
-- WRONG: consumer sees 0 rows
begin;
  select pgque.send('orders', '{"order_id": 1}'::jsonb);
  select pgque.force_next_tick('orders');
  select pgque.ticker();
  select * from pgque.receive('orders', 'processor', 100);  -- 0 rows
commit;
The asymmetry is: receive → process → ack belongs in one transaction (for exactly-once effects on the same database), but send → tick → receive requires committed boundaries between each step. Go (pgxpool) and TypeScript (pg.Pool) satisfy this transparently since each query runs in its own implicit transaction. Python requires explicit commits between send and the consumer side; the high-level Consumer class handles this internally.

​
Three-table TRUNCATE rotation

Each queue owns three event tables. At any given time, one is the “hot” table receiving new events. When the rotation period elapses (or maint() triggers it), PgQue rotates the tables:
  1. The hot table becomes the “middle” table (still queryable by active batches)
  2. The oldest table is TRUNCATEd — removing all events in one operation with no dead tuples
  3. A fresh empty table becomes the new hot table
This is why n_dead_tup = 0 across event tables even under sustained load with a blocked xmin horizon. VACUUM never needs to reclaim dead tuples from event tables because there are none.

​
Consumer loop

The underlying PgQ consumer loop: 
batch_id = next_batch(queue, consumer)   -- NULL → sleep and retry
events   = get_batch_events(batch_id)
process(events)                           -- event_retry per event on failure
finish_batch(batch_id)
commit
pgque.receive() = next_batch() + get_batch_events(). pgque.ack() = finish_batch(). pgque.nack() = event_retry() (with DLQ routing when retry_count >= max_retries).

​
Glossary

One row in a queue table. Delivered at-least-once. Columns: ev_id, ev_time, ev_txid (xid8), ev_retry, ev_type, ev_data, ev_extra1..4.
The set of events between two consecutive ticks, served to a consumer together. All events in a batch share the same batch_id.
A named event stream. Backed by 3 rotating tables, purged by TRUNCATE. Any number of queues can coexist in one database.
A position marker in the event stream. Delimits batch boundaries. Contains a PostgreSQL snapshot that determines which events are visible in the batch.
The component that creates ticks. In PgQue’s default pg_cron path: one 1-second cron slot calls pgque.ticker_loop(), which invokes pgque.ticker() every tick_period_ms milliseconds (100 ms / 10 ticks/sec by default).
Subscribes to a queue, reads batches, calls ack or nack. Any number of consumers can subscribe to the same queue; each has its own cursor and independently sees every event (fan-out by default).
The periodic TRUNCATE-and-swap cycle that keeps event tables bloat-free. Managed by pgque.maint() (step 1) and the pgque_rotate_step2 cron job (step 2).

Build docs developers (and LLMs) love

Get started for freeTalk to us