Changelog - Walrus

This page documents the evolution of Walrus from a single-node WAL library to a distributed message streaming platform.

Version 0.3.0

This is the current major release introducing the distributed streaming platform.

Distributed Message Streaming Platform

Walrus 0.3.0 represents a major architectural evolution, transforming the single-node WAL engine into a fault-tolerant distributed system.

Raft Consensus

Integrated Raft consensus (Octopii) for metadata coordination across cluster nodes.

Segment-Based Sharding

Topics split into segments with automatic leadership rotation for load balancing.

Lease-Based Write Fencing

Only designated leaders can write to segments, preventing split-brain scenarios.

Automatic Rollover

Monitor loop triggers segment rollover when threshold exceeded (~1M entries).

New Features

TCP Client Protocol

Simple length-prefixed text protocol for producers and consumers:

Commands:
  REGISTER <topic>       → Create topic if missing
  PUT <topic> <payload>  → Append to topic
  GET <topic>            → Read next entry
  STATE <topic>          → Get topic metadata
  METRICS                → Get Raft metrics

Interactive CLI

New walrus-cli binary for cluster interaction:

cargo run --bin walrus-cli -- --addr 127.0.0.1:9091

Comprehensive Test Suite

Integration tests for distributed scenarios:

cluster-test-logs - Basic smoke test
cluster-test-rollover - Segment rollover
cluster-test-resilience - Node failure recovery
cluster-test-recovery - Cluster restart persistence
cluster-test-stress - Concurrent writes
cluster-test-multi-topic - Multiple topics

Architecture Components

Node Controller

Routes client requests to appropriate segment leaders, manages write leases synchronized every 100ms, tracks logical offsets for rollover detection.

Raft Engine (Octopii)

Maintains consensus for metadata changes only (not data path). Handles leader election and log replication across nodes.

Cluster Metadata

Raft state machine storing topic → segment → leader mappings, sealed segment counts, and node addresses.

Storage Engine

Wraps Walrus engine with lease-based write fencing. Only accepts writes if node holds lease for that segment.

Configuration

New environment variables:

Variable	Default	Description
`WALRUS_MAX_SEGMENT_ENTRIES`	`1000000`	Entries before rollover
`WALRUS_MONITOR_CHECK_MS`	`10000`	Monitor loop interval

Breaking Changes

Version 0.3.0 introduces a new distributed architecture. Applications using the standalone library (0.2.x) can continue using walrus-rust crate unchanged. Distributed features require migration to the new cluster architecture.

Version 0.2.0

Batch Operations and io_uring

This release focused on performance improvements through batch operations and asynchronous I/O.

New Features

Atomic Batch Writes

let entries = vec![
    b"entry1".to_vec(),
    b"entry2".to_vec(),
    b"entry3".to_vec(),
];
wal.batch_append_for_topic("my-topic", &entries)?;

Performance Benefit

Batch operations reduce syscall overhead and enable io_uring optimizations on Linux.

Batch Read Operations

let entries = wal.batch_read_for_topic("my-topic", 10, true)?;
for entry in entries {
    println!("Read: {:?}", String::from_utf8_lossy(&entry.data));
}

io_uring Support

On Linux, Walrus can leverage io_uring for asynchronous batched I/O operations, significantly improving throughput.

Disable io_uring with the WALRUS_DISABLE_IO_URING environment variable if compatibility issues arise.

Dual Storage Backends

Two backend implementations:

FD Backend

Uses pread/pwrite syscalls for file operations. Compatible with all platforms.

Mmap Backend

Memory-mapped I/O for improved read performance. Default on most platforms.

Namespace Isolation

New _for_key constructors for multi-tenancy:

let wal1 = Walrus::new_for_key("tenant1")?;
let wal2 = Walrus::new_for_key("tenant2")?;

Improvements

Fsync Modes

Added FsyncSchedule::SyncEach for durability and FsyncSchedule::NoFsync for maximum throughput.

let config = WalConfig {
    fsync_schedule: FsyncSchedule::SyncEach,
    ..Default::default()
};

Enhanced Documentation

Comprehensive architecture documentation, design docs, and API reference improvements.

Expanded Benchmarks

Added benchmarks for batch operations comparing Walrus against Kafka and RocksDB.

Bug Fixes

Fixed: Tail read offset tracking in concurrent scenarios where multiple readers access the same topic
Fixed: Edge case in cursor advancement when segment boundaries align with read batch sizes

Version 0.1.0

Initial Release

The first public release of Walrus as a standalone Write-Ahead Log library.

Core Features

WAL Functionality

Append-only log storage with sequential write guarantees.

Topic-Based Organization

Logical separation of data streams into named topics.

Consistency Modes

Configurable fsync policies for durability vs. throughput tradeoffs.

Persistent Cursors

Read offset tracking survives process restarts.

API Example

use walrus_rust::{Walrus, ReadConsistency};

// Create a new WAL instance
let wal = Walrus::new()?;

// Write data to a topic
wal.append_for_topic("my-topic", b"Hello, Walrus!")?;

// Read data from the topic
if let Some(entry) = wal.read_next("my-topic", true)? {
    println!("Read: {:?}", String::from_utf8_lossy(&entry.data));
}

Benchmarks

Initial benchmark suite comparing Walrus against industry-standard systems:

RocksDB Write-Ahead Log
Apache Kafka (single broker)

See Performance Benchmarks for detailed results.

Memory-Mapped I/O

Implemented mmap-based backend for improved read performance on sequential workloads.

Future Roadmap

Planned features for upcoming releases:

Multi-Segment Writes

Allow concurrent writes to multiple segments within a topic for higher aggregate throughput.

Compression

Optional compression for sealed segments to reduce storage overhead.

Read-Ahead Caching

Predictive caching for sequential read workloads to improve latency.

Replication

Data-level replication across nodes for durability (currently metadata-only).

Feature roadmap is subject to change based on community feedback and project priorities. See Contributing to help shape future development.

Version Support

Version	Status	Support
0.3.x	Current	Active development
0.2.x	Maintenance	Security fixes only
0.1.x	End of Life	No longer supported

Users on versions 0.1.x should upgrade to 0.2.x or later for continued support.

Getting Started

Core Concepts

Standalone Library

Distributed Cluster

Operations

Resources

Documentation Index

​Version 0.3.0

​Distributed Message Streaming Platform

Raft Consensus

Segment-Based Sharding

Lease-Based Write Fencing

Automatic Rollover

​New Features

​TCP Client Protocol

​Interactive CLI

​Comprehensive Test Suite

​Architecture Components

​Configuration

​Breaking Changes

​Version 0.2.0

​Batch Operations and io_uring

​New Features

​Atomic Batch Writes

Performance Benefit

​Batch Read Operations

​io_uring Support

​Dual Storage Backends

FD Backend

Mmap Backend

​Namespace Isolation

​Improvements

​Bug Fixes

​Version 0.1.0

​Initial Release

​Core Features

WAL Functionality

Topic-Based Organization

Consistency Modes

Persistent Cursors

​API Example

​Benchmarks

​Memory-Mapped I/O

​Future Roadmap

Multi-Segment Writes

Compression

Read-Ahead Caching

Replication

​Version Support

Build docs developers (and LLMs) love

Version 0.3.0

Distributed Message Streaming Platform

New Features

TCP Client Protocol

Interactive CLI

Comprehensive Test Suite

Architecture Components

Configuration

Breaking Changes

Version 0.2.0

Batch Operations and io_uring

New Features

Atomic Batch Writes

Batch Read Operations

io_uring Support

Dual Storage Backends

Namespace Isolation

Improvements

Bug Fixes

Version 0.1.0

Initial Release

Core Features

API Example

Benchmarks

Memory-Mapped I/O

Future Roadmap

Version Support