Records

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/s2-streamstore/s2/llms.txt

Use this file to discover all available pages before exploring further.

​

Record structure

pub struct SequencedRecord {
    pub position: StreamPosition,  // seq_num + timestamp
    pub record: Record,            // The actual data
}

​

Position

{
  "seq_num": 42,
  "timestamp": 1709481600000
}

• seq_num: Unique sequence number (0-based, incrementing)
• timestamp: Unix timestamp in milliseconds

​

Headers

{
  "headers": [
    {"name": "content-type", "value": "application/json"},
    {"name": "user-id", "value": "123"}
  ]
}

• Names must not be empty (except for command records)
• Both names and values are arbitrary bytes
• Headers contribute to metered size and storage

​

Body

{
  "body": "SGVsbG8sIFdvcmxkIQ=="  // Base64-encoded binary
}

​

Record types

pub enum RecordType {
    Command = 1,
    Envelope = 2,
}

​

Envelope records

• Contain zero or more headers with non-empty names
• Contain a body with arbitrary data
• Used for storing events, messages, logs, etc.

{
  "seq_num": 100,
  "timestamp": 1709481600000,
  "headers": [
    {"name": "event-type", "value": "user.created"},
    {"name": "trace-id", "value": "abc123"}
  ],
  "body": "eyJ1c2VyX2lkIjoxMjN9"  // {"user_id":123}
}

​

Command records

• Have exactly one header with an empty name
• The header value identifies the command type
• The body contains the command payload

​

Fence command

{
  "headers": [{"name": "", "value": "fence"}],
  "body": "bXktdW5pcXVlLXRva2Vu"  // "my-unique-token"
}

• Must be ≤ 48 bytes in length
• Start as an empty string by default
• Can be any arbitrary bytes
• Enable exactly-once semantics

pub const MAX_FENCING_TOKEN_LENGTH: usize = 48;

​

Trim command

{
  "headers": [{"name": "", "value": "trim"}],
  "body": "AAAAAAAAAGY="  // SeqNum: 100 (8 bytes, big-endian)
}

• Body must be exactly 8 bytes (u64 sequence number)
• Records before the trim point become unreadable
• The trim point itself can never be trimmed
• Used for implementing retention policies

​

Data encoding formats

pub enum Format {
    Raw,      // UTF-8 strings
    Base64,   // Base64-encoded binary
}

​

Raw format (default)

• Header names, values, and body are represented as UTF-8 strings
• Efficient for text data
• Storage is in UTF-8 encoding

{
  "headers": [
    {"name": "content-type", "value": "text/plain"}
  ],
  "body": "Hello, World!"
}

​

Base64 format

• Header names, values, and body are Base64-encoded
• Safe for binary data transmission
• Efficient storage of binary data (stored as original bytes)

{
  "headers": [
    {"name": "Y29udGVudC10eXBl", "value": "aW1hZ2UvcG5n"}
  ],
  "body": "iVBORw0KGgoAAAANSUhEUgAAAAUA..."
}

curl https://api.s2.dev/streams/my-stream/records \
  -H "S2-Basin: my-basin" \
  -H "S2-Format: base64"

​

Metered size

• Billing (for s2.dev)
• Batch limits (max 1 MiB per batch)
• Storage tracking

impl MeteredSize for Record {
    fn metered_size(&self) -> usize {
        8 + (match self {
            Record::Command(command) => 
                2 + command.op().to_id().len() + command.payload().len(),
            Record::Envelope(envelope) => 
                (2 * envelope.headers().len())
                + envelope.headers().deep_size()
                + envelope.body().len()
        })
    }
}

• 8 bytes overhead per record
• 2 bytes per header (delimiter overhead)
• Header deep size: Sum of all header name and value lengths
• Body length: Size of the body in bytes

​

Record encoding

​

Magic byte

pub struct MagicByte {
    pub record_type: RecordType,      // 3 bits
    pub metered_size_varlen: u8,      // 2 bits (1-3 bytes)
}

• Bits 0-2: Record type (1=Command, 2=Envelope)
• Bits 3-4: Metered size variable length (0=1 byte, 1=2 bytes, 2=3 bytes)

​

Wire format

+-------------+------------------+------------------+
| Magic Byte  | Metered Size     | Record Data      |
| (1 byte)    | (1-3 bytes)      | (variable)       |
+-------------+------------------+------------------+

+----------------+------------------+-------------+
| Header Count   | Headers          | Body        |
| (2 bytes)      | (variable)       | (variable)  |
+----------------+------------------+-------------+

+-------------+-------------+-------------+-------------+
| Name Length | Name        | Value Length| Value       |
| (2 bytes)   | (variable)  | (2 bytes)   | (variable)  |
+-------------+-------------+-------------+-------------+

​

Appending records

curl -X POST https://api.s2.dev/streams/my-stream/records \
  -H "S2-Basin: my-basin" \
  -H "Content-Type: application/json" \
  -d '{
    "records": [
      {
        "timestamp": 1709481600000,
        "headers": [
          {"name": "event-type", "value": "order.created"}
        ],
        "body": "eyJvcmRlcl9pZCI6MTIzfQ=="
      }
    ]
  }'

{
  "start": {"seq_num": 100, "timestamp": 1709481600000},
  "end": {"seq_num": 101, "timestamp": 1709481600000},
  "tail": {"seq_num": 101, "timestamp": 1709481600000}
}

​

Batch constraints

• Minimum: 1 record per batch
• Maximum: 1000 records per batch
• Size limit: Total metered size ≤ 1 MiB per batch

​

Append options

• Client-provided timestamp in milliseconds
• Behavior depends on stream timestamping configuration
• Server enforces monotonicity

• Enforce that first record gets this sequence number
• Fails with seq-num-mismatch if tail has moved
• Enables optimistic concurrency control

• Enforce the current fencing token matches
• Fails with fencing-token-mismatch if token differs
• Enables exactly-once semantics

​

Reading records

curl https://api.s2.dev/streams/my-stream/records?seq_num=100&count=10 \
  -H "S2-Basin: my-basin"

{
  "records": [
    {
      "seq_num": 100,
      "timestamp": 1709481600000,
      "headers": [
        {"name": "event-type", "value": "order.created"}
      ],
      "body": "eyJvcmRlcl9pZCI6MTIzfQ=="
    }
  ],
  "tail": {"seq_num": 150, "timestamp": 1709482000000}
}

​

Record validation

• Headers with empty names are only allowed for command records
• Command records must have exactly one header
• Fencing tokens must be ≤ 48 bytes
• Trim points must be 8 bytes (u64)
• Total batch size must be ≤ 1 MiB metered bytes

​

Best practices

1. Use headers for metadata: Store searchable metadata in headers
2. Choose appropriate format: Use raw for text, base64 for binary
3. Batch appends: Send multiple records per request for better throughput
4. Include timestamps: Provide client timestamps when event time matters
5. Leverage fencing: Use fencing tokens for exactly-once critical operations
6. Monitor metered size: Large records impact performance and costs

​

Next steps