Client Protocol

Protocol Overview

Distributed Walrus exposes a simple, length-prefixed text protocol over TCP. Clients can connect to any node in the cluster, and the system handles routing automatically.

Design Principles

Text-based: Commands are UTF-8 strings (human-readable, easy to debug)
Length-prefixed: Fixed 4-byte header prevents message ambiguity
Stateless: Each command is independent (except read cursors are server-side)
Synchronous: Request-response model, one command at a time per connection

The protocol is inspired by Redis RESP but simplified for streaming log workloads.

Wire Format

Request Format

All requests follow this structure:

┌─────────────────┬──────────────────────────────┐
│  Length (4B)    │  UTF-8 Command Text          │
│  Little-Endian  │  (variable length)           │
└─────────────────┴──────────────────────────────┘

Example: PUT logs hello

Bytes:  0x0E 0x00 0x00 0x00  P  U  T     l  o  g  s     h  e  l  l  o
        └────────┬─────────┘  └────────────────┬──────────────────────┘
           Length = 14              "PUT logs hello" (14 bytes)

Response Format

Responses use the same length-prefixed format:

┌─────────────────┬──────────────────────────────┐
│  Length (4B)    │  UTF-8 Response Text         │
│  Little-Endian  │  (variable length)           │
└─────────────────┴──────────────────────────────┘

Success responses:

OK - Command succeeded
OK <payload> - Command succeeded with data
EMPTY - No data available (for GET)

Error responses:

ERR <message> - Command failed with error message

Command Reference

REGISTER

Create a topic if it doesn’t already exist. Idempotent. Syntax:

REGISTER <topic>

Parameters:

topic: Topic name (alphanumeric, no spaces)

Response:

OK - Topic created or already exists
ERR <message> - Failed to create topic

Example:

$ walrus-cli register logs
OK

Under the hood:

Node checks if topic exists in metadata
If missing, proposes CreateTopic via Raft consensus
Initial leader is selected via consistent hashing
Metadata is replicated across all nodes

PUT

Append data to a topic. Automatically routes to the current segment leader. Syntax:

PUT <topic> <payload>

Parameters:

topic: Topic name
payload: Data to append (rest of command after topic, can contain spaces)

Response:

OK - Data appended successfully
ERR unknown topic - Topic doesn’t exist (use REGISTER first)
ERR NotLeaderForPartition - Temporary leadership transfer error (retry)

Example:

$ walrus-cli put logs "User login: alice@example.com"
OK

$ walrus-cli put logs "{\"event\":\"purchase\",\"amount\":99.99}"
OK

Flow:

Client sends PUT to any node (e.g., Node 2)
Node 2 checks metadata: which node leads this topic’s current segment?
If Node 2 is leader: write directly to local Walrus
If Node 1 is leader: forward via internal RPC to Node 1
Leader checks write lease and appends to Walrus
Leader tracks entry count for rollover detection
Response flows back to client

Large payloads (>64KB) will be rejected. For bulk ingestion, send multiple smaller PUTs or consider batching at the application layer.

GET

Read the next entry from a topic using a shared server-side cursor. The cursor automatically advances across sealed segments. Syntax:

GET <topic>

Parameters:

topic: Topic name

Response:

OK <data> - Entry data (cursor advances)
EMPTY - No data available yet (cursor doesn’t advance)
ERR unknown topic - Topic doesn’t exist

Example:

# First read
$ walrus-cli get logs
OK hello world

# Second read
$ walrus-cli get logs
OK another message

# No more data
$ walrus-cli get logs
EMPTY

Cursor behavior: The server maintains a shared cursor per topic:

struct ReadCursor {
    segment: u64,              // Current segment (1, 2, 3, ...)
    delivered_in_segment: u64, // Entries delivered from this segment
}

Cursor advancement rules:

Within segment: Increment delivered_in_segment on each successful read
Sealed segment exhausted: If delivered_in_segment >= sealed_count, move to next segment
Active segment empty: Return EMPTY (wait for more data)

Example cursor progression:

Initial state:        {segment: 1, delivered: 0}
After 1st GET:        {segment: 1, delivered: 1}
After 1,000,000 GETs: {segment: 1, delivered: 1,000,000}
(Segment 1 sealed at 1M entries)
Next GET:             {segment: 2, delivered: 0}

The cursor is shared across all client connections. Multiple consumers reading from the same topic will receive different entries (round-robin style).

STATE

Retrieve metadata for a topic, including segment layout and leadership. Syntax:

STATE <topic>

Parameters:

topic: Topic name

Response:

JSON object with topic state
ERR unknown topic - Topic doesn’t exist

Example:

$ walrus-cli state logs

{
  "current_segment": 3,
  "leader_node": 2,
  "last_sealed_entry_offset": 1950000,
  "sealed_segments": {
    "1": 1000000,
    "2": 950000
  },
  "segment_leaders": {
    "1": 3,
    "2": 1,
    "3": 2
  }
}

Field descriptions:

Field	Type	Description
`current_segment`	`u64`	Active segment accepting writes
`leader_node`	`u64`	Node ID currently leading the active segment
`last_sealed_entry_offset`	`u64`	Cumulative entries across all sealed segments
`sealed_segments`	`map<u64, u64>`	Segment ID → entry count for sealed segments
`segment_leaders`	`map<u64, u64>`	Segment ID → node ID (historical leaders)

Use cases:

Monitor segment distribution across nodes
Debug routing issues
Calculate total entries: last_sealed_entry_offset + (entries in current segment)
Understand leadership history

METRICS

Retrieve Raft cluster metrics and health information. Syntax:

METRICS

Response:

JSON object with Raft metrics

Example:

$ walrus-cli metrics

{
  "id": 1,
  "current_term": 5,
  "current_leader": 1,
  "state": "Leader",
  "last_log_index": 1234,
  "last_applied": 1234,
  "membership_config": {
    "voters": [1, 2, 3],
    "learners": []
  }
}

Key metrics:

Metric	Description
`current_leader`	Node ID of Raft leader (or null if no leader)
`state`	Node’s Raft role: `Leader`, `Follower`, `Candidate`
`last_log_index`	Latest Raft log entry index
`last_applied`	Last index applied to state machine
`voters`	List of voting node IDs

A healthy cluster should show:

current_leader is non-null
last_applied closely trails last_log_index
voters includes all expected nodes

Connection Management

Persistent Connections

Clients can reuse the same TCP connection for multiple commands:

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('127.0.0.1', 9091))

# Send multiple commands over same connection
send_command(sock, "REGISTER logs")
send_command(sock, "PUT logs message1")
send_command(sock, "PUT logs message2")
send_command(sock, "GET logs")

sock.close()

Benefits:

Avoid TCP handshake overhead
Lower latency for burst operations
Connection pooling friendly

Timeouts and Retries

Network Errors
Application Errors

Connection refused: Node is down or port not open

Retry with a different node (e.g., 9092, 9093)

Connection reset: Node crashed mid-request

Reconnect and retry command (PUTs are NOT idempotent)

ERR NotLeaderForPartition: Lease is transferring

Wait 100-200ms and retry

ERR no raft leader known: Cluster is re-electing

Wait and retry (usually resolves in <5 seconds)

ERR unknown topic: Topic doesn’t exist

Send REGISTER first, then retry

Load Balancing

For high availability, use a load balancer to distribute client connections:

┌──────────┐
│  Clients │
└────┬─────┘
     │
┌────▼──────────┐
│ Load Balancer │
│   (HAProxy)   │
└───┬──┬───┬────┘
    │  │   │
┌───▼──▼───▼────┐
│ Node 1  Node 2│
│    Node 3     │
└───────────────┘

HAProxy config example:

listen walrus-cluster
    bind *:9090
    mode tcp
    balance roundrobin
    server node1 192.168.1.10:9091 check
    server node2 192.168.1.11:9092 check
    server node3 192.168.1.12:9093 check

Clients connect to :9090, and HAProxy distributes to :9091-9093.

Client Libraries

Official CLI

The Rust CLI client is the reference implementation:

# Interactive REPL
cargo run --bin walrus-cli -- --addr 127.0.0.1:9091

# One-off commands
cargo run --bin walrus-cli -- --addr 127.0.0.1:9091 put logs "data"

Source: distributed-walrus/src/bin/walrus-cli.rs (see GitHub)

Third-Party Clients

Community clients (contribution welcome!):

Python: pip install walrus-client (planned)
Go: github.com/user/walrus-go (planned)
Node.js: npm install walrus-js (planned)

Writing a Custom Client

Minimal client in 50 lines:

import struct
import socket

class WalrusClient:
    def __init__(self, host='127.0.0.1', port=9091):
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.sock.connect((host, port))
    
    def _send(self, command):
        data = command.encode('utf-8')
        length = struct.pack('<I', len(data))
        self.sock.sendall(length + data)
        
        resp_len = struct.unpack('<I', self.sock.recv(4))[0]
        response = self.sock.recv(resp_len).decode('utf-8')
        return response
    
    def register(self, topic):
        return self._send(f"REGISTER {topic}")
    
    def put(self, topic, data):
        return self._send(f"PUT {topic} {data}")
    
    def get(self, topic):
        resp = self._send(f"GET {topic}")
        if resp == "EMPTY":
            return None
        return resp[3:]  # Strip "OK "
    
    def close(self):
        self.sock.close()

# Usage
client = WalrusClient()
client.register("events")
client.put("events", "user_signup")
print(client.get("events"))  # "user_signup"
client.close()

Protocol Limitations

Current limitations (may be addressed in future versions):

No authentication: All connections are trusted
No TLS: Data sent in plaintext
No batching: One command per request
No streaming: Must poll with GET for new data
Single cursor: No consumer groups or parallel consumers per topic
No explicit offsets: Cannot seek to arbitrary position

For production deployments, consider:

Running in a private network with firewall rules
Using a VPN or SSH tunnel for encryption
Implementing authentication at the load balancer layer

Next Steps

Segment Management

Learn how segments roll over and leases work

Failure Recovery

Handle errors and node failures

Getting Started

Core Concepts

Standalone Library

Distributed Cluster

Operations

Resources

Client Protocol

Protocol Overview

Design Principles

Wire Format

Request Format

Response Format

Command Reference

REGISTER

PUT

GET

STATE

METRICS

Connection Management

Persistent Connections

Timeouts and Retries

Load Balancing

Client Libraries

Official CLI

Third-Party Clients

Writing a Custom Client

Protocol Limitations

Next Steps

Segment Management

Failure Recovery

Build docs developers (and LLMs) love

Getting Started

Core Concepts

Standalone Library

Distributed Cluster

Operations

Resources

Documentation Index

​Protocol Overview

​Design Principles

​Wire Format

​Request Format

​Response Format

​Command Reference

​REGISTER

​PUT

​GET

​STATE

​METRICS

​Connection Management

​Persistent Connections

​Timeouts and Retries

​Load Balancing

​Client Libraries

​Official CLI

​Third-Party Clients

​Writing a Custom Client

​Protocol Limitations

​Next Steps

Segment Management

Failure Recovery

Build docs developers (and LLMs) love

Protocol Overview

Design Principles

Wire Format

Request Format

Response Format

Command Reference

REGISTER

PUT

GET

STATE

METRICS

Connection Management

Persistent Connections

Timeouts and Retries

Load Balancing

Client Libraries

Official CLI

Third-Party Clients

Writing a Custom Client

Protocol Limitations

Next Steps