Process Submission Webhook — QStash Code Execution

Buildml offloads code execution to a background queue so that the browser is never blocked waiting for a Docker container to finish running tests. When a user clicks Run or Submit, a tRPC mutation publishes a message to Upstash QStash, which then delivers it to this webhook for processing.

POST /api/webhooks/process-submission is not a public endpoint. It is called exclusively by Upstash QStash. Every incoming request must carry a valid QStash HMAC signature in the upstash-signature header — any request without a verified signature is immediately rejected with 401 Unauthorized. Do not attempt to call this endpoint directly from client code or external services.

Endpoint

POST /api/webhooks/process-submission

Security

QStash signs every delivery with an HMAC using your signing keys. The webhook verifies the signature via the @upstash/qstash Receiver before any payload is parsed:

// src/lib/qstash.ts
import { Client, Receiver } from "@upstash/qstash";

export const qstash = new Client({ token: env.QSTASH_TOKEN });

export const receiver = new Receiver({
  currentSigningKey: env.QSTASH_CURRENT_SIGNING_KEY,
  nextSigningKey: env.QSTASH_NEXT_SIGNING_KEY,
});

The webhook reads the raw request body as text before parsing JSON, because the HMAC is computed over the raw bytes:

const bodyText = await req.text();
const isValid = await receiver.verify({ body: bodyText, signature });

if (!isValid) {
  return new NextResponse("Invalid signature", { status: 401 });
}

Environment Variable	Purpose
`QSTASH_TOKEN`	Authenticates the publisher (`qstash.publishJSON`)
`QSTASH_CURRENT_SIGNING_KEY`	Verifies the most-recently-rotated QStash signature
`QSTASH_NEXT_SIGNING_KEY`	Verifies the upcoming key during key rotation

Message Types

The webhook supports two message types, determined by the type field in the JSON body.

RUN

A RUN message is published by submission.run (tRPC mutation). It executes user code and stores the result in Redis for 10 minutes so the client can poll for it. Payload:

{
  "type": "RUN",
  "runId": "550e8400-e29b-41d4-a716-446655440000",
  "problemId": "clx4m2r0p0000abc123def456",
  "code": "def solution(n):\n    return n * 2",
  "userId": "clx4m2r0p0001abc123def456"
}

type

"RUN"

required

Discriminator field. Must be the string "RUN".

runId

string

required

A crypto.randomUUID() UUID generated by the tRPC mutation. Used as the Redis key suffix: run_result:{runId}.

problemId

string

required

The Prisma cuid of the problem being run. The webhook fetches the problem record (including its parent problemSet) to resolve the executor slugs.

code

string

required

The user’s Python source code to execute.

userId

string

required

The authenticated user’s ID. Included for observability and future per-user metrics.

SUBMIT

A SUBMIT message is published by submission.submit (tRPC mutation). It grades the code permanently and updates the Submission database row. Payload:

{
  "type": "SUBMIT",
  "submissionId": "clx4m2r0p0002abc123def456",
  "userId": "clx4m2r0p0001abc123def456"
}

type

"SUBMIT"

required

Discriminator field. Must be the string "SUBMIT".

submissionId

string

required

The Prisma cuid of the Submission row created by the tRPC mutation before publishing to QStash. The webhook fetches this record to retrieve the stored code and problem context.

userId

string

required

The authenticated user’s ID.

Processing Flow

RUN Flow

Signature verification

The webhook reads the raw body and verifies the upstash-signature header using the QStash Receiver. A missing or invalid signature returns 401 immediately.

Fetch problem from database

The webhook queries Prisma for the problem matching problemId, including the related problemSet, to obtain problem.slug and problem.problemSet.slug.

const problem = await prisma.problem.findUnique({
  where: { id: problemId },
  include: { problemSet: true },
});

Call the executor

The webhook POSTs to {EXECUTOR_URL}/execute with the user’s code and the resolved slugs. The x-secret header authenticates the call.

{
  "code": "<user's Python code>",
  "task_id": "<problem.slug>",
  "problem_set_slug": "<problemSet.slug>"
}

Parse executor response

The executor response is converted into a status (PASS, FAIL, or ERROR) and a human-readable output string. See Result Parsing below.

Store result in Redis

The result is written to the key run_result:{runId} with a 600-second (10-minute) TTL. The client polls submission.getStatus with the runId to retrieve it.

await redis.set(
  `run_result:${runId}`,
  { status, output, passed, total, results },
  { ex: 600 },
);

SUBMIT Flow

Signature verification

Identical to the RUN flow — the QStash signature is verified before any database work begins.

Fetch submission and problem from database

The webhook fetches the Submission row (which contains the user’s stored code) and eagerly includes the Problem and its parent ProblemSet.

const submission = await prisma.submission.findUnique({
  where: { id: submissionId },
  include: {
    problem: {
      include: { problemSet: true },
    },
  },
});

Call the executor

Same executor call as RUN — the code comes from submission.code rather than the message payload.

Parse executor response

The executor response is converted to a final status and output. See Result Parsing below.

Update Submission row

The Submission record is updated with the final status and output. The submission is now queryable through submission.getStatus.

await prisma.submission.update({
  where: { id: submissionId },
  data: { status, output },
});

Executor Request & Response

Request

POST {EXECUTOR_URL}/execute
Content-Type: application/json
x-secret: {EXECUTOR_SECRET}

{
  "code": "<Python source code>",
  "task_id": "<problem slug>",
  "problem_set_slug": "<problem set slug>"
}

Response

{
  "passed": 3,
  "total": 4,
  "results": [
    { "name": "test_basic", "passed": true },
    { "name": "test_edge_case", "passed": true },
    { "name": "test_large_input", "passed": true },
    { "name": "test_negative", "passed": false, "error": "AssertionError: expected -1, got 0" }
  ],
  "stdout": "",
  "stderr": "",
  "error": null
}

passed

number

Number of test cases that passed.

total

number

Total number of test cases executed.

results

array

Per-test breakdown. Each entry has name (string), passed (boolean), and an optional error string describing the failure.

stdout

string

Captured standard output from the user’s code.

stderr

string

Captured standard error output. Appended to the output string if non-empty.

error

string | null

Top-level error message if the executor itself failed (e.g. syntax error, timeout). When set, status is forced to ERROR regardless of test counts.

Result Parsing

The webhook maps the executor response to three possible statuses:

Status	Condition
`PASS`	`result.passed === result.total` and `result.total > 0`
`FAIL`	Any tests failed (`passed < total`)
`ERROR`	`result.error` is set, or the executor returned a non-2xx HTTP status

The human-readable output string is assembled as follows:

N/M tests passed
  ✓ test_basic
  ✓ test_edge_case
  ✓ test_large_input
  ✗ test_negative: AssertionError: expected -1, got 0

--- stderr ---
Traceback (most recent call last): ...

The format rules are:

First line is always {passed}/{total} tests passed
Each test result follows as ✓ {name} (pass) or ✗ {name}: {error} (fail)
If result.error is set, it is appended after a blank line
If result.stderr is non-empty, a --- stderr --- section is appended

Error Handling

If any step throws — database lookup fails, executor is unreachable, or the executor returns a non-2xx status — the webhook catches the error and writes an ERROR status so the client is never left polling indefinitely. For SUBMIT errors, the Submission row is updated:

await prisma.submission.update({
  where: { id: submissionId },
  data: { status: "ERROR", output: errorMessage },
});

For RUN errors, the Redis key is written with status: "ERROR" (still with the 600-second TTL):

await redis.set(
  `run_result:${runId}`,
  { status: "ERROR", output: errorMessage, passed: 0, total: 0, results: [] },
  { ex: 600 },
);

The webhook always returns 500 in the error case so QStash can retry delivery according to its retry policy.

Polling for Results

After receiving a runId or submissionId from the tRPC mutation, clients poll the submission.getStatus query:

import { api } from "~/trpc/react";

const { data } = api.submission.getStatus.useQuery(
  { runId },
  {
    // Poll every second until we have a final result
    refetchInterval: (query) =>
      query.state.data?.status === "PENDING" ? 1000 : false,
  },
);

getStatus returns { status: "PENDING", output: null } when the Redis key does not yet exist, so the client can distinguish “not started” from “processing” from “done”.

Overview

tRPC Routers

Webhooks

Process Submission Webhook — QStash Code Execution

Endpoint

Security

Message Types

RUN

SUBMIT

Processing Flow

RUN Flow

SUBMIT Flow

Executor Request & Response

Request

Response

Result Parsing

Error Handling

Polling for Results

Build docs developers (and LLMs) love

Overview

tRPC Routers

Webhooks

Documentation Index

​Endpoint

​Security

​Message Types

​RUN

​SUBMIT

​Processing Flow

​RUN Flow

​SUBMIT Flow

​Executor Request & Response

​Request

​Response

​Result Parsing

​Error Handling

​Polling for Results

Build docs developers (and LLMs) love

Endpoint

Security

Message Types

RUN

SUBMIT

Processing Flow

RUN Flow

SUBMIT Flow

Executor Request & Response

Request

Response

Result Parsing

Error Handling

Polling for Results