Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/denoland/celld/llms.txt

Use this file to discover all available pages before exploring further.

celld shuts down gracefully on SIGTERM or SIGINT. These are the signals that systemctl stop, docker stop, and a Kubernetes pod deletion send by default. During shutdown, celld hands every resident cell it owns to a peer before exiting, so no acknowledged write is lost and no cell is left without an owner on a live node. A peer takes over each released cell immediately — there is no takeover gap of the kind an abrupt kill -9 would cause.

Graceful shutdown sequence

1

Node receives SIGTERM or SIGINT

The signal is delivered by the orchestrator (systemctl stop, docker stop, Kubernetes pod delete) or manually by an operator.
2

Health endpoint reports 503

/__celld/health immediately begins returning 503 Service Unavailable. A load balancer that polls this endpoint removes the node from rotation before new application traffic arrives.
3

New requests receive 503

Any new incoming request receives a 503 response and the connection is closed, encouraging the client to retry on a healthy node. Requests that were already in flight continue to be served.
4

Resident cells are released

celld releases ownership of each resident cell so that peers can take over immediately. A cell that is actively serving a request is handed off when that request finishes. Up to CELLD_RELEASES releases run concurrently (default: 128).
5

Node exits

The node exits once all handoff work and in-flight requests have finished, or when CELLD_SHUTDOWN_DRAIN_MS milliseconds have elapsed — whichever comes first. An idle node with no resident cells exits immediately.
The internal listener continues to accept /state requests during the drain. The response includes occupied, evicting, and restoring counters so operators can observe progress in real time.

Drain configuration

Two environment variables control the drain behaviour:
VariableDefaultDescription
CELLD_RELEASES128Maximum number of concurrent cell releases during shutdown. Keeps a node with many cells from flooding the object store at exit.
CELLD_SHUTDOWN_DRAIN_MS25000Maximum time in milliseconds the drain is allowed to run. The node exits when this deadline passes even if not all handoffs have finished.
Set CELLD_SHUTDOWN_DRAIN_MS to a value below the stop-grace timeout of your orchestrator — for example, systemd TimeoutStopSec or Kubernetes terminationGracePeriodSeconds. If the drain deadline exceeds the orchestrator’s grace period, the orchestrator will send SIGKILL before celld can finish, defeating the graceful shutdown. A safe rule of thumb is to set CELLD_SHUTDOWN_DRAIN_MS to at least a few seconds less than the orchestrator limit.

Rolling updates

celld has no built-in rollout command. Instead it relies on the health endpoint to let your orchestrator pace the rollout safely:
1

Deploy the new version to the fleet bucket

Run celld deploy to publish the updated Worker code. Running nodes pick up the new deployment automatically on the next request — you do not need to restart nodes just to update application code.
2

Stop one node

Signal the first node with SIGTERM (via systemctl stop, kubectl rollout, or your orchestrator’s rolling-update mechanism). The node enters the drain sequence described above.
3

Start the replacement node

Start a new node running the updated celld binary. Wait for its health endpoint to return {"ok":true} with HTTP 200 before proceeding.
4

Verify restoring=0

Use celld diagnose and confirm that the replacement node reports restoring=0. This means all cold cell activations from the restart have finished, and the node is fully ready to accept owned cells from the next restart.
5

Repeat for each node

Move to the next node in the fleet and repeat until all nodes run the new binary.

Version upgrade notes

The upgrade from v0.1.0 to v0.2.0 must NOT be performed as a rolling update. You must stop every v0.1.0 node before starting any v0.2.0 node. A fleet must never mix these two versions.Two incompatible changes make a mixed fleet unsafe:
  • v0.2.0 nodes advertise the internal listener address in ownership records. v0.1.0 peers cannot interpret these records, so they cannot contact a v0.2.0 owner.
  • v0.2.0 compacts replicated data into block objects that a v0.1.0 reader cannot restore.
To upgrade: stop all v0.1.0 nodes, verify the fleet is fully drained, then start the v0.2.0 nodes.
For all other version upgrades, check the release notes for any similar hard-cutover requirements before choosing a rolling update strategy.

Operator API for shutdown

The internal listener exposes an alpha operator API for programmatic control. Endpoints are subject to change — keep operator tooling and the celld binary at the same release.

Start a graceful shutdown

curl -X POST http://10.0.0.12:8081/shutdown
POST /shutdown on the internal listener starts the same graceful handoff sequence that SIGTERM triggers: the health endpoint reports 503, new requests receive 503, resident cells are released, and the node exits when the drain finishes.

Check node state during drain

curl http://10.0.0.12:8081/state
The /state response reports occupied (current resident cells), evicting (cells being released), and restoring (cold activations in progress). The internal listener continues to serve /state throughout the drain period.

Same-node reload (preserve mode)

curl -X POST http://10.0.0.12:8081/shutdown?handoff=preserve
POST /shutdown?handoff=preserve prepares a clean in-place restart. In preserve mode, celld keeps the ownership records for resident cells rather than releasing them to peers. This is used when restarting the same process on the same node — for example, to reload a configuration change — without triggering a full ownership handoff across the fleet. The internal listener still serves /state during the preserve-mode drain, so a supervisor process can poll the node state and start the replacement process only after the drain completes.

Build docs developers (and LLMs) love