Compression - Cowrie

Cowrie Gen2 supports transparent payload compression using Gzip or Zstd algorithms. Compression is applied to the dictionary and root value, reducing network transfer and storage costs.

Overview

Compression in Cowrie is optional and automatic: when enabled, the encoder compresses payloads and the decoder transparently decompresses them.

Wire Format

Header (4 bytes):
  Magic:   'S' 'J' (0x53 0x4A)
  Version: 0x02
  Flags:   [Compressed][CompressionType][...]

If Compressed flag set:
  OrigLen:  uvarint (original uncompressed size)
  Payload:  compressed(DictLen + Dict + RootValue)

If not compressed:
  DictLen:  uvarint
  Dict:     ...
  RootValue: ...

Compression Algorithms

Gzip

Standard deflate compression, widely supported.

import "github.com/Neumenon/cowrie"

opts := cowrie.EncodeOptions{
    Compression: cowrie.CompressionGzip,
}
data, err := cowrie.EncodeWithOptions(value, opts)

Characteristics:

Compression ratio: 2-5x for text, 1.2-2x for mixed data
Speed: Medium (10-50 MB/s encode, 100-300 MB/s decode)
Use case: Standard HTTP, broad compatibility

Zstd

Zstandard by Facebook, faster with better compression.

opts := cowrie.EncodeOptions{
    Compression: cowrie.CompressionZstd,
}
data, err := cowrie.EncodeWithOptions(value, opts)

Characteristics:

Compression ratio: 2-6x for text, 1.3-2.5x for mixed data
Speed: Fast (100-400 MB/s encode, 400-1200 MB/s decode)
Use case: Modern APIs, internal services, high-throughput systems

None

Disable compression (default for backward compatibility).

opts := cowrie.EncodeOptions{
    Compression: cowrie.CompressionNone,  // default
}

Flag Encoding

Compression flags are bit-packed in byte 3 of the header:

Bit(s)	Meaning
0	Compressed (0=no, 1=yes)
1-2	Type (00=none, 01=gzip, 10=zstd)
3	Column hints present
4-7	Reserved

Examples:

0x00: No compression
0x03: Compressed with Gzip (bits 0=1, 1-2=01)
0x05: Compressed with Zstd (bits 0=1, 1-2=10)

When to Use Compression

Enable Compression For

Text-Heavy Payloads: JSON-like objects with repeated keys
Large Arrays: String arrays, repeated structures
Mixed Data: Text + numbers (2-4x compression)
Network Transfer: API responses, remote storage
Cold Storage: S3, archives, backups

Disable Compression For

Already Compressed Data: JPEG images, Opus audio, compressed tensors
Small Payloads: < 256 bytes (overhead > savings)
Low Latency: Real-time systems where CPU matters more than bandwidth
Pre-Compressed Transport: HTTPS with Content-Encoding already enabled

Compression Decision Tree

Is payload > 256 bytes?
  No → Disable compression (overhead too high)
  Yes → Continue

Contains mostly binary data (images, audio, compressed tensors)?
  Yes → Disable compression (already compressed)
  No → Continue

Is latency critical (< 1ms)?
  Yes → Disable compression (CPU cost matters)
  No → Enable Zstd compression

Automatic Compression

The encoder automatically decides whether to compress:

// If compressed size >= original size, keep uncompressed
if len(compressed) >= len(original) {
    return original, CompressionNone
}
return compressed, CompressionZstd

This ensures compression never increases payload size.

Compression Examples

Example 1: Text-Heavy Payload

// Large object with repeated keys
data := make(map[string]any)
for i := 0; i < 1000; i++ {
    data[fmt.Sprintf("user_%d", i)] = map[string]any{
        "name":  fmt.Sprintf("User %d", i),
        "email": fmt.Sprintf("user%d@example.com", i),
        "age":   20 + i%50,
        "verified": i%2 == 0,
    }
}

// Without compression
uncompressed, _ := cowrie.Encode(cowrie.FromAny(data))
fmt.Println("Uncompressed:", len(uncompressed))  // ~80KB

// With Zstd
opts := cowrie.EncodeOptions{Compression: cowrie.CompressionZstd}
compressed, _ := cowrie.EncodeWithOptions(cowrie.FromAny(data), opts)
fmt.Println("Compressed:", len(compressed))      // ~15KB
fmt.Println("Ratio:", float64(len(uncompressed))/float64(len(compressed)))  // ~5.3x

Example 2: Mixed Payload

// Query response with embeddings and metadata
response := map[string]any{
    "ids":      []string{"doc1", "doc2", "doc3"},
    "docs":     []string{"Hello world", "Foo bar", "Test data"},
    "scores":   []float32{0.95, 0.87, 0.72},
    "embeddings": make([]float32, 768*3),  // 3 embeddings
    "metadata": map[string]any{
        "took_ms": 42,
        "total":   1000,
        "offset":  0,
    },
}

// Zstd compression
opts := cowrie.EncodeOptions{Compression: cowrie.CompressionZstd}
data, _ := cowrie.EncodeWithOptions(cowrie.FromAny(response), opts)

// Text compresses ~3x, float32 arrays ~1.2x, overall ~2x

Example 3: Image Payload (Don’t Compress)

// JPEG image (already compressed)
jpegData, _ := os.ReadFile("photo.jpg")  // 1MB
image := cowrie.Image(cowrie.ImageFormatJPEG, 1920, 1080, jpegData)

// Without compression
uncompressed, _ := cowrie.Encode(image)
fmt.Println("Size:", len(uncompressed))  // ~1.00MB + 7 bytes

// With compression (WASTED CPU!)
opts := cowrie.EncodeOptions{Compression: cowrie.CompressionZstd}
compressed, _ := cowrie.EncodeWithOptions(image, opts)
fmt.Println("Size:", len(compressed))  // ~1.00MB + overhead (no savings!)

Lesson: Disable compression for pre-compressed data.

Security: Decompression Bombs

Cowrie protects against decompression bombs where a small compressed payload expands to gigabytes of RAM.

Protection Mechanism

const MaxDecompressedSize = 256 * 1024 * 1024  // 256 MB

func decompressGzip(data []byte) ([]byte, error) {
    r, _ := gzip.NewReader(bytes.NewReader(data))
    
    // Limit decompression
    limited := io.LimitReader(r, MaxDecompressedSize+1)
    out, _ := io.ReadAll(limited)
    
    if len(out) > MaxDecompressedSize {
        return nil, cowrie.ErrDecompressedTooLarge
    }
    return out, nil
}

Configure Limits

opts := codec.MasterReaderOptions{
    MaxDecompressedSize: 100 * 1024 * 1024,  // 100 MB
}
reader := codec.NewMasterReader(data, opts)

Default: 256 MB for Gen2 document decoding, 100 MB for master stream frames.

Attack Example

// Attacker creates 1KB payload that decompresses to 1GB
malicious := compress(strings.Repeat("a", 1_000_000_000))  // 1KB compressed

// Decoder rejects
_, err := cowrie.Decode(malicious)
// err == cowrie.ErrDecompressedTooLarge

Compression in Master Streams

Master streams automatically compress individual frames when beneficial:

opts := codec.MasterWriterOptions{
    Compression: cowrie.CompressionZstd,
}
writer := codec.NewMasterWriter(stream, opts)

// Each frame compressed independently
for _, record := range records {
    writer.Write(record)  // Auto-compressed if size > 256 bytes
}

Frame-Level Compression

Each frame has its own compression decision:

Frame 1: Small (100 bytes) → Not compressed
Frame 2: Large text (10KB) → Compressed 3x
Frame 3: Image (1MB) → Not compressed (JPEG)
Frame 4: Mixed (50KB) → Compressed 2x

This provides optimal compression without wasting CPU on pre-compressed data.

Performance Benchmarks

Gzip

Payload Type	Size	Compress Time	Decompress Time	Ratio
Text (JSON-like)	100KB	20ms	5ms	4.2x
Mixed (text+numbers)	500KB	100ms	20ms	2.8x
Float32 array	1MB	50ms	15ms	1.3x
JPEG image	2MB	120ms	40ms	1.02x

Zstd

Payload Type	Size	Compress Time	Decompress Time	Ratio
Text (JSON-like)	100KB	5ms	2ms	4.8x
Mixed (text+numbers)	500KB	25ms	8ms	3.2x
Float32 array	1MB	12ms	4ms	1.5x
JPEG image	2MB	30ms	10ms	1.03x

Takeaway: Zstd is 3-5x faster with slightly better compression.

Best Practices

1. Default to Zstd

// Good: Modern, fast, efficient
opts := cowrie.EncodeOptions{
    Compression: cowrie.CompressionZstd,
}

2. Skip for Pre-Compressed Data

// Check if payload contains images/audio
if containsMedia(data) {
    opts.Compression = cowrie.CompressionNone
}

3. Profile Your Workload

// Measure compression benefit
uncompressed, _ := cowrie.Encode(data)
compressed, _ := cowrie.EncodeWithOptions(data, opts)

ratio := float64(len(uncompressed)) / float64(len(compressed))
if ratio < 1.2 {
    // < 20% savings, disable compression
    opts.Compression = cowrie.CompressionNone
}

4. Deterministic + Compression

// For content addressing, enable both
opts := cowrie.EncodeOptions{
    Deterministic: true,
    Compression:   cowrie.CompressionZstd,
}
data, _ := cowrie.EncodeWithOptions(value, opts)
hash := sha256.Sum256(data)  // Stable hash

5. Network vs Storage

// Network: Prioritize speed
networkOpts := cowrie.EncodeOptions{
    Compression: cowrie.CompressionZstd,  // Fast
}

// Cold storage: Prioritize size
storageOpts := cowrie.EncodeOptions{
    Compression: cowrie.CompressionGzip,  // Smaller
}

HTTP Integration

Server

func handler(w http.ResponseWriter, r *http.Request) {
    // Check Accept-Encoding
    acceptZstd := strings.Contains(r.Header.Get("Accept-Encoding"), "zstd")
    
    opts := cowrie.EncodeOptions{}
    if acceptZstd {
        opts.Compression = cowrie.CompressionZstd
        w.Header().Set("Content-Encoding", "cowrie-zstd")
    }
    
    data, _ := cowrie.EncodeWithOptions(response, opts)
    w.Header().Set("Content-Type", "application/cowrie")
    w.Write(data)
}

Client

func fetch(url string) (*cowrie.Value, error) {
    req, _ := http.NewRequest("GET", url, nil)
    req.Header.Set("Accept", "application/cowrie")
    req.Header.Set("Accept-Encoding", "zstd, gzip")
    
    resp, _ := http.DefaultClient.Do(req)
    defer resp.Body.Close()
    
    data, _ := io.ReadAll(resp.Body)
    
    // Cowrie decoder handles compression automatically
    return cowrie.Decode(data)
}

Compression Ratio by Data Type

Data Type	Gzip Ratio	Zstd Ratio
JSON-like text	3-5x	4-6x
String arrays	2-4x	3-5x
Repeated objects	3-6x	4-7x
Float32 tensors	1.1-1.3x	1.2-1.5x
Int64 arrays	1.2-1.8x	1.4-2.0x
JPEG images	1.0-1.02x	1.0-1.03x
PNG images	1.0-1.1x	1.0-1.15x
Audio (Opus, AAC)	1.0-1.05x	1.0-1.05x

Troubleshooting

Compression Not Applied

Symptom: Payload size unchanged after enabling compression. Causes:

Payload < 256 bytes (below threshold)
Already compressed data (images, audio)
Compression increased size (automatic fallback)

Solution: Check compressed vs uncompressed size:

fmt.Println("Compressed size:", len(compressed))
fmt.Println("Flag:", compressed[3])  // Check bit 0

Decompression Too Slow

Symptom: High CPU usage during decode. Causes:

Using Gzip (slower than Zstd)
Unnecessarily compressing binary data

Solution: Switch to Zstd or disable compression:

// Measure decode time
start := time.Now()
cowrie.Decode(data)
fmt.Println("Decode time:", time.Since(start))

ErrDecompressedTooLarge

Symptom: Decode fails with “decompressed too large” error. Causes:

Legitimate large payload exceeding limits
Malicious decompression bomb

Solution: Increase limit or investigate payload:

// Check original length in frame header
origLen := binary.LittleEndian.Uint32(data[4:8])
fmt.Println("Original size:", origLen)

// Increase limit if legitimate
opts := codec.MasterReaderOptions{
    MaxDecompressedSize: 500 * 1024 * 1024,  // 500 MB
}

Streaming - Compress streamed frames
Security Limits - Configure decompression limits
ML Types - When NOT to compress tensors
Graph Types - Compress graph shards

Getting Started

Core Concepts

Language SDKs

Advanced Features

CLI Tool

Performance

Documentation Index

​Overview

​Wire Format

​Compression Algorithms

​Gzip

​Zstd

​None

​Flag Encoding

​When to Use Compression

​Enable Compression For

​Disable Compression For

​Compression Decision Tree

​Automatic Compression

​Compression Examples

​Example 1: Text-Heavy Payload

​Example 2: Mixed Payload

​Example 3: Image Payload (Don’t Compress)

​Security: Decompression Bombs

​Protection Mechanism

​Configure Limits

​Attack Example

​Compression in Master Streams

​Frame-Level Compression

​Performance Benchmarks

​Gzip

​Zstd

​Best Practices

​1. Default to Zstd

​2. Skip for Pre-Compressed Data

​3. Profile Your Workload

​4. Deterministic + Compression

​5. Network vs Storage

​HTTP Integration

​Server

​Client

​Compression Ratio by Data Type

​Troubleshooting

​Compression Not Applied

​Decompression Too Slow

​ErrDecompressedTooLarge

​Related Topics

Build docs developers (and LLMs) love

Overview

Wire Format

Compression Algorithms

Gzip

Zstd

None

Flag Encoding

When to Use Compression

Enable Compression For

Disable Compression For

Compression Decision Tree

Automatic Compression

Compression Examples

Example 1: Text-Heavy Payload

Example 2: Mixed Payload

Example 3: Image Payload (Don’t Compress)

Security: Decompression Bombs

Protection Mechanism

Configure Limits

Attack Example

Compression in Master Streams

Frame-Level Compression

Performance Benchmarks

Gzip

Zstd

Best Practices

1. Default to Zstd

2. Skip for Pre-Compressed Data

3. Profile Your Workload

4. Deterministic + Compression

5. Network vs Storage

HTTP Integration

Server

Client

Compression Ratio by Data Type

Troubleshooting

Compression Not Applied

Decompression Too Slow

ErrDecompressedTooLarge

Related Topics