Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/denoland/celld/llms.txt

Use this file to discover all available pages before exploring further.

celld diagnose reads the node leases from the fleet bucket and sends a signed direct probe to each live peer. It does not acquire a lease, does not take ownership of any cell, and does not change state — it is a read-only inspection tool.

​
Running diagnose

Point celld diagnose at your bucket using the same credential variables you use to start nodes: 
celld diagnose \
  --bucket "$CELLD_BUCKET" \
  --endpoint "$S3_ENDPOINT" \
  --region "$AWS_REGION"
To restrict the probe to specific nodes, pass --peer NODE_ID one or more times: 
celld diagnose \
  --bucket "$CELLD_BUCKET" \
  --endpoint "$S3_ENDPOINT" \
  --region "$AWS_REGION" \
  --peer node-a \
  --peer node-b

​
What it reports

For each node lease found in the bucket, celld diagnose attempts a direct probe and reports:
  • Expired records — node leases whose TTL has passed and that no live node has renewed
  • Malformed or unsafe advertised addresses — addresses that are structurally invalid or that contain a literal public IP without CELLD_UNSAFE_PUBLIC_ADVERTISE=1
  • Unreachable peers — nodes whose advertised address did not respond within the probe deadline
  • Authentication failures — probes that reached a node but failed HMAC verification, indicating a mismatched peer-auth secret or a clock that is too far skewed
  • Protocol version mismatches — nodes running a different internal protocol version that is incompatible with the probing node
Each node line in the output also includes the following live counters sampled from the probed process:
FieldMeaning
restoringCold routes currently holding an activation permit or waiting for one
resident_cellsCells currently resident in memory
websocketsOpen WebSocket connections
rss_bytesResident set size of the node process in bytes
cpu_percentCPU utilisation sample
fdsOpen and limit file-descriptor counts
pressuredWhether the node is currently under memory pressure
shed_cellsNumber of cells shed due to pressure

​
The restoring field and rolling updates

The restoring value counts each cold route that holds an activation permit or is waiting for one. Because a capacity waiter already holds a permit, each cold route is counted exactly once — there is no double counting. During a rolling update you must wait for every node to report restoring=0 before restarting the next node. This ensures that the cold work that began on the current node finishes before the next restart removes more warm capacity from the fleet.
A node with restoring > 0 is still activating cells from cold storage. Restarting it before that number reaches zero shortens the warm-capacity window for the fleet and forces those cells to activate from scratch on another node.
A safe rolling-update loop looks like this: 
# For each node in the fleet:
#   1. Send SIGTERM to the node and wait for shutdown to complete.
#   2. Start the replacement node; wait for /__celld/health to return 200.
#   3. Poll /state on the replacement node until restoring=0.
#   4. Move to the next node.

​
Internal operator API

The internal listener exposes an unauthenticated operator API. All paths below are on the --internal-listen address.
PathDescription
/stateReports current occupancy, eviction, and restoration values. Remains available throughout a graceful shutdown drain.
/cell/NAMEResolves or activates a named cell for an operator check.
/evict/NAMEEvicts a resident cell.
/do/NAMESends a direct Durable Object request to the named cell.
POST /shutdownStarts a graceful ownership handoff, identical to SIGTERM.
POST /shutdown?handoff=preservePrepares a clean same-node reload and keeps the ownership records intact.
GET /__celld/probeServes the signed diagnostic probe consumed by celld diagnose. Also available during graceful shutdown drain.
A /state response looks like this: 
{
  "occupied": 42,
  "evicting": 0,
  "restoring": 3
}
The operator API has no authentication. Any client that can reach the internal listener can inspect node state, activate or evict cells, or stop the process. Restrict access with a firewall rule or a private network overlay. Do not expose the internal listener to the public internet.
The operator API is an alpha interface. A release can change its paths or response formats without notice. Keep your operator tooling and the celld release at the same version.

Build docs developers (and LLMs) love

Get started for freeTalk to us