Documentation Index
Fetch the complete documentation index at: https://mintlify.com/msolli/proletarian/llms.txt
Use this file to discover all available pages before exploring further.
Proletarian guarantees that every enqueued job will run at least once. It goes to great lengths to ensure that no jobs are silently lost due to exceptions, network errors, database failures, or process crashes. The flip side of this guarantee is that in rare failure scenarios a job may run more than once, which means your handler functions must be written to tolerate duplicate execution.
How the Guarantee Works
When Proletarian processes a job, it wraps the entire operation in a database transaction:
- The job row is locked with
SELECT … FOR UPDATE SKIP LOCKED.
- Your handler function runs.
- On success, the job is deleted from the queue table and inserted into the
archived_job table with a :success status — both in the same commit.
Because the removal from the queue and the archival happen atomically, a job that hasn’t been archived will always remain visible to future polls. If the worker process crashes mid-execution, the transaction rolls back and another worker thread (on the same or a different machine) will pick up the job and try again.
The Failure Scenario That Causes Duplicate Execution
The at-least-once guarantee exists because of a narrow window of failure: the handler has finished successfully, but the database transaction that removes the job from the queue and writes the archive record cannot be committed — for example, because the database went offline at precisely that moment.
[Worker] [Database]
│ │
├── BEGIN TRANSACTION ─────────────►│
├── SELECT job FOR UPDATE ─────────►│
│◄── job data ──────────────────────┤
│ │
├── handler runs successfully │
│ │
├── DELETE job ────────────────────►│
├── INSERT archived_job ───────────►│
├── COMMIT ───────────────────────►│
│ ← database offline
│◄── connection error ──────────────┤
│
[job is still in queue — will be retried]
The Outbox Pattern: Transactional Enqueueing
One of Proletarian’s most powerful features is the ability to enqueue a job inside the same database transaction as your business logic. This is known as the Outbox Pattern.
Consider sending a confirmation email when a user registers. Without transactional enqueueing, you might write the user to the database and then enqueue the job as separate steps — meaning the job could be lost if the process crashes between the two. With Proletarian, both happen atomically:
(ns your-app.handlers
(:require [next.jdbc :as jdbc]
[proletarian.job :as job]))
(defn register-user! [system request]
(jdbc/with-transaction [tx (:db system)]
;; Write the new user record
(let [user (db/insert-user! tx (parse-user-data (:body request)))]
;; Enqueue the confirmation email in the same transaction.
;; If the transaction rolls back, the job is never created.
;; If the transaction commits, the job is guaranteed to be processed.
(job/enqueue! tx ::send-confirmation-email
{:user-id (:id user)
:email (:email user)
:first-name (:first-name user)})
;; Return the response
{:status 201 :body (user->response user)})))
Writing Idempotent Handlers
Idempotence means that executing an operation multiple times with the same inputs produces the same result as executing it once. Here are practical strategies for achieving idempotence in your job handlers.
Strategy 1: Use the Job ID as an Idempotency Key
Every job has a stable UUID assigned at enqueue time, available as :proletarian.job/job-id in advanced handler mode. You can record this UUID when the job executes and skip re-execution if it has been seen before:
(defmethod handle-job! ::charge-subscription
[{:proletarian.job/keys [job-id payload]}]
;; Check if we've already processed this job
(when-not (db/idempotency-key-exists? job-id)
(billing/charge! (:customer-id payload) (:amount payload))
;; Record that we've processed this job-id
(db/record-idempotency-key! job-id)))
The job ID UUID is stable across retry attempts. The same UUID is used for every retry of a given job, making it a reliable idempotency key for deduplication.
Strategy 2: Database UPSERT Semantics
Instead of checking then inserting, use INSERT … ON CONFLICT DO NOTHING (PostgreSQL) or INSERT IGNORE (MySQL). The database enforces uniqueness, and duplicate executions simply become no-ops:
(defmethod handle-job! ::sync-external-record
[_job-type {:keys [record-id data]}]
;; UPSERT: create or update — safe to run multiple times
(db/upsert-external-record! {:id record-id
:data data
:synced-at (Instant/now)}))
Strategy 3: Check Entity Status Before Acting
Model your entities with status fields and guard the handler behind a status check. Only proceed if the entity is in the expected state:
(defmethod handle-job! ::fulfill-order
[_job-type {:keys [order-id]}]
(let [order (db/fetch-order order-id)]
;; Only fulfill if the order is still in "pending" state.
;; If we've already fulfilled it, this becomes a safe no-op.
(when (= :pending (:status order))
(fulfillment/ship-order! order)
(db/update-order-status! order-id :fulfilled))))
