Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/nubskr/walrus/llms.txt

Use this file to discover all available pages before exploring further.

​
What is Walrus as a Library?

Walrus can be embedded directly into your Rust application as a high-performance Write-Ahead Log (WAL) library. This gives you complete control over data persistence, consistency guarantees, and storage configuration without running a separate distributed cluster.

​
When to Use Library Mode

Use Walrus as a library when:
  • Single Process: Your application runs in a single process and doesn’t need distributed coordination
  • Embedded Storage: You need a reliable, fast WAL for state management, event sourcing, or data pipelines
  • Custom Control: You want fine-grained control over consistency models, fsync schedules, and storage backends
  • Low Latency: You need sub-millisecond write and read operations without network overhead

​
When to Use Distributed Mode

Consider distributed Walrus when:
  • Multiple Processes: Your workload spans multiple machines or processes
  • High Availability: You need replication and failover capabilities
  • Horizontal Scaling: Your throughput requirements exceed a single machine
  • Remote Access: Clients need network access to the log

​
Key Features

​
High Performance

  • Optimized for concurrent reads and writes
  • Block-based storage with 10MB blocks (1GB files)
  • Support for io_uring on Linux for parallel I/O (FD backend)
  • Minimal allocation overhead

​
Topic-Based Organization

use walrus_rust::Walrus;

let wal = Walrus::new()?;

// Each topic maintains independent read/write streams
wal.append_for_topic("orders", b"order-123")?;
wal.append_for_topic("payments", b"payment-456")?;
wal.append_for_topic("inventory", b"item-789")?;
Topics provide logical separation within a single WAL instance. Each topic has:
  • Independent read cursors
  • Separate write streams
  • Isolated checkpoint positions

​
Configurable Consistency

Choose the right trade-off between durability and performance: 
use walrus_rust::{Walrus, ReadConsistency};

// Strictly-at-once: every checkpoint persisted immediately
let wal = Walrus::with_consistency(ReadConsistency::StrictlyAtOnce)?;

// At-least-once: checkpoint every N reads for higher throughput
let wal = Walrus::with_consistency(
    ReadConsistency::AtLeastOnce { persist_every: 1000 }
)?;
See Configuration for detailed consistency model explanations.

​
Dual Storage Backends

Walrus supports two storage implementations: FD Backend (Default)
  • Uses file descriptors with pread/pwrite syscalls
  • Batch operations use io_uring on Linux for parallel I/O
  • Supports O_SYNC mode for synchronous writes
  • Best for: Linux systems, batch-heavy workloads
Mmap Backend
  • Memory-mapped file I/O
  • Sequential batch operations (no io_uring)
  • Cross-platform compatibility
  • Best for: Non-Linux platforms, Windows
See Storage Backends for backend selection and performance characteristics.

​
Persistent Read Offsets

Read positions automatically survive process restarts: 
{
    let wal = Walrus::new()?;
    wal.append_for_topic("events", b"event-1")?;
    wal.append_for_topic("events", b"event-2")?;
    
    // Checkpoint=true persists the read position
    let entry = wal.read_next("events", true)?.unwrap();
    assert_eq!(entry.data, b"event-1");
} // WAL dropped, process could restart here

{
    let wal = Walrus::new()?;
    // Resumes from persisted position
    let entry = wal.read_next("events", true)?.unwrap();
    assert_eq!(entry.data, b"event-2");
}

​
Namespace Isolation

Create isolated WAL instances with separate storage directories: 
use walrus_rust::Walrus;

// Each namespace has its own directory under wal_files/
let tenant1 = Walrus::new_for_key("tenant-123")?;
let tenant2 = Walrus::new_for_key("tenant-456")?;

tenant1.append_for_topic("orders", b"data-1")?;
tenant2.append_for_topic("orders", b"data-2")?;

// Data is completely isolated between namespaces
Storage layout: 
wal_files/
├── tenant-123/
│   ├── data files
│   └── read_offset_idx.db
└── tenant-456/
    ├── data files
    └── read_offset_idx.db

​
Basic Usage

use walrus_rust::{Walrus, ReadConsistency, FsyncSchedule};

fn main() -> std::io::Result<()> {
    // Create a WAL instance
    let wal = Walrus::new()?;
    
    // Write data to topics
    wal.append_for_topic("my-topic", b"Hello, Walrus!")?;
    
    // Read with checkpoint (consumes the entry)
    if let Some(entry) = wal.read_next("my-topic", true)? {
        println!("Data: {:?}", String::from_utf8_lossy(&entry.data));
    }
    
    // Peek without consuming (checkpoint=false)
    if let Some(entry) = wal.read_next("my-topic", false)? {
        println!("Peeking: {:?}", String::from_utf8_lossy(&entry.data));
    }
    
    Ok(())
}
When you need explicit control over the data directory, use the builder API to avoid race conditions with the WALRUS_DATA_DIR environment variable: 
use walrus_rust::Walrus;
use std::path::PathBuf;

let wal = Walrus::builder()
    .data_dir(PathBuf::from("/var/lib/myapp/wal"))
    .key("tenant-123")
    .build()?;
This is especially important in multi-threaded applications where different threads need different data directories.

​
Performance Characteristics

  • Block size: 10MB per block
  • File size: 100 blocks per file (1GB files)
  • Batch limits: Up to 2,000 entries or ~10GB payload per batch
  • Default fsync interval: 200ms
  • Checksum algorithm: FNV-1a 64-bit for data integrity

​
Environment Variables

​
WALRUS_DATA_DIR
string
default:"./wal_files"
Storage directory for all WAL data
​
WALRUS_INSTANCE_KEY
string
Default namespace key for all instances
​
WALRUS_QUIET
string
Set to 1 to suppress debug output
Example usage: 
export WALRUS_DATA_DIR=/var/lib/myapp/wal
export WALRUS_INSTANCE_KEY=production
export WALRUS_QUIET=1

​
Next Steps

Build docs developers (and LLMs) love

Get started for freeTalk to us