Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/denoland/celld/llms.txt

Use this file to discover all available pages before exploring further.

A celld fleet is any number of nodes that all point at the same bucket. Nodes discover each other through the ownership leases and node-lease objects that each node writes to that bucket. There is no join command, no seed list, and no fixed membership configuration. To grow the fleet, start another node with the same bucket settings. To shrink it, stop a node gracefully and its cells move to the remaining nodes automatically.

​
Adding a node

Start each node with the same bucket settings. Give each internal listener a different address that the other nodes in the fleet can reach, and set --advertise to that address. The nodes find each other through the leases in the bucket: 
# Node A
celld \
  --bucket "$CELLD_BUCKET" \
  --endpoint "$S3_ENDPOINT" \
  --region "$AWS_REGION" \
  --listen 0.0.0.0:8080 \
  --internal-listen 10.0.0.10:8081 \
  --advertise node-a.internal:8081

# Node B — same bucket, different internal address
celld \
  --bucket "$CELLD_BUCKET" \
  --endpoint "$S3_ENDPOINT" \
  --region "$AWS_REGION" \
  --listen 0.0.0.0:8080 \
  --internal-listen 10.0.0.11:8081 \
  --advertise node-b.internal:8081
Both nodes watch the same bucket. When Node B starts, it reads the existing node leases and discovers Node A. When Node A needs to hand off a cell, it knows Node B’s advertised address and can contact it directly.
Never expose the internal listener port to the public internet. The internal listener carries an unauthenticated operator API in addition to peer coordination traffic. Restrict access to your private or overlay network.

​
How peers find each other

The bucket is the discovery and authority layer. Each running node writes a lease object to the bucket that records its advertised address and a session ID. Other nodes read those leases to build their view of the fleet membership. The bucket does not provide network reachability — that is the responsibility of your network configuration. Once a node has an address from the bucket, it contacts that peer directly over HTTP. Peer HTTP requests are protected by multiple security layers:
  • Version field — rejects requests from nodes running a different protocol version.
  • Body signature — detects tampering or corruption in transit.
  • HMAC — authenticates that the sender holds the shared fleet secret (see Peer authentication below).
  • Clock limit — rejects requests with a timestamp too far from the receiver’s clock.
  • Replay protection — prevents a captured request from being re-used.
Put the internal (advertised) addresses on a private network you control, or use an encrypted overlay such as WireGuard or Tailscale. celld does not terminate TLS on its own listeners.

​
Cell placement

celld has no central placement controller. When a request arrives for a cell that no node currently owns (an inactive or released cell), whichever node receives the request acquires ownership by writing an ownership record to the bucket. Normal traffic therefore distributes cells across nodes organically as requests arrive. One 8 GB node holds approximately 1,000 resident cells. Admission is controlled at two levels:
  • CELLD_MAX_RESIDENT_CELLS — a hard ceiling on resident cells enforced at admission. A request that would exceed this limit receives a 503 until capacity is available.
  • CELLD_ACTIVATIONS — the limit for concurrent cold-cell activations (default: available CPU count or 128, whichever is smaller). Cold activations involve restoring SQLite data from the bucket; this limit prevents a burst of cold starts from saturating I/O.

​
Memory pressure

When a node approaches its memory limit, celld sheds resident cells gracefully to keep the process healthy:
  • CELLD_MAX_RSS_MB — the RSS threshold for pressure shedding. Default is 80% of the available memory on the host; set to 0 to disable RSS-based shedding entirely.
Under memory pressure celld selects the least-recently-used idle cells and releases them in order:
1

Replicate durably

The cell’s SQLite data is replicated to the bucket so no acknowledged write is lost.
2

Fence the cell

The ownership record is invalidated so no stale peer can serve the cell’s data.
3

Release

The cell transitions to the inactive state. Any node in the fleet can activate it on the next request.
CELLD_OUTPUT_GATE (default 1) ensures RPO=0: celld does not acknowledge a write to the application until the replication write to the bucket has succeeded. Setting this to 0 removes the replication wait and accepts the possibility of losing an acknowledged write on node failure.

​
Peer authentication

The first node to start a fleet creates a fleet/peer-auth.json object in the bucket. This object contains the shared HMAC secret used to authenticate all peer requests. Every subsequent node reads this secret at startup. All inter-node HTTP requests carry an HMAC signature derived from this secret, a request timestamp, and a nonce, providing:
  • Authentication — only nodes that can read the bucket can forge a valid request.
  • Clock-bounded validity — a request is rejected if the sender’s clock differs from the receiver’s by more than the allowed skew.
  • Replay protection — a nonce prevents a captured request from being submitted twice.
Because the bucket access credential is equivalent to full fleet administrator access, protect your bucket credentials with the same care as any fleet secret. The bucket contains deployments, SQLite replicas, ownership records, node leases, and the peer-authentication secret.

​
Diagnose a fleet

Use celld diagnose to inspect a running fleet without acquiring any ownership. It reads the node leases from the bucket and probes each live peer: 
celld diagnose \
  --bucket "$CELLD_BUCKET" \
  --endpoint "$S3_ENDPOINT" \
  --region "$AWS_REGION"
The report shows:
  • Expired lease records
  • Unsafe or incorrect advertised addresses
  • Peers that cannot be reached
  • Authentication failures
  • Protocol-version mismatches
Each node line includes a restoring counter — the number of cold cell activations currently in progress on that node. During a rolling update you should wait for every node to report restoring=0 before restarting the next node, so that the cold-start work from one restart finishes before the next restart removes more warm capacity.

Build docs developers (and LLMs) love

Get started for freeTalk to us