Encode & Decode

Overview

Cowrie Gen2 provides efficient binary encoding with dictionary-coded keys, optional compression, and forward-compatible extension types.

Encoding Functions

Encode

func Encode(v *Value) ([]byte, error)

Encodes a Cowrie value to Gen2 binary format.

*Value

required

The value to encode

data

[]byte

Encoded binary data with Gen2 header and dictionary

error

Error if encoding fails (e.g., unsupported type)

Wire format:

Magic:   'S' 'J'     (2 bytes)
Version: 0x02        (1 byte)
Flags:   0x00        (1 byte)
DictLen: uvarint
Dict:    DictLen × [len:uvarint][utf8 bytes]
RootVal: encoded value

Example:

obj := cowrie.Object(
    cowrie.Member{Key: "name", Value: cowrie.String("Alice")},
    cowrie.Member{Key: "age", Value: cowrie.Int64(30)},
)

data, err := cowrie.Encode(obj)
if err != nil {
    log.Fatal(err)
}
// data contains Gen2 binary with dictionary ["name", "age"]

EncodeAppend

func EncodeAppend(dst []byte, v *Value) ([]byte, error)

Encodes a value and appends to an existing buffer. Useful for avoiding allocations when managing buffers manually.

dst

[]byte

required

Destination buffer to append to

*Value

required

Value to encode

result

[]byte

Buffer with encoded data appended

Example:

buf := make([]byte, 0, 1024)
buf, err := cowrie.EncodeAppend(buf, cowrie.Int64(42))

EncodeTo

func EncodeTo(w io.Writer, v *Value) error

Encodes a value directly to an io.Writer.

io.Writer

required

Writer to output encoded data

*Value

required

Value to encode

Example:

file, _ := os.Create("data.cowrie")
defer file.Close()
err := cowrie.EncodeTo(file, myValue)

EncodeWithOptions

func EncodeWithOptions(v *Value, opts EncodeOptions) ([]byte, error)

Encodes a value with explicit options for deterministic output and null omission.

*Value

required

Value to encode

opts

EncodeOptions

required

Encoding options

EncodeOptions fields:

Deterministic

bool

Sorts object keys lexicographically for reproducible output. Essential for hashing, caching, and content addressing.

OmitNull

bool

Excludes null values from encoded objects. Useful for sparse data and reducing payload size.

Example:

opts := cowrie.EncodeOptions{
    Deterministic: true,  // Stable output for content hashing
    OmitNull: true,       // Skip null fields
}
data, err := cowrie.EncodeWithOptions(myValue, opts)

EncodeWithHints

func EncodeWithHints(v *Value, hints []ColumnHint) ([]byte, error)

Encodes a value with column hints for columnar optimization. See ColumnReader for details.

*Value

required

Value to encode

hints

[]ColumnHint

required

Column hints for selective field access

Example:

hints := []cowrie.ColumnHint{
    cowrie.NewHint("id", cowrie.HintInt64, cowrie.HintFlagRequired),
    cowrie.NewTensorHint("embedding", cowrie.HintFloat32, []int{128}, cowrie.HintFlagColumnar),
}
data, err := cowrie.EncodeWithHints(arrayValue, hints)

Decoding Functions

Decode

func Decode(data []byte) (*Value, error)

Decodes Gen2 binary data into a Cowrie value. Uses default security limits to prevent memory exhaustion.

data

[]byte

required

Encoded binary data

value

*Value

Decoded Cowrie value

error

Error if decoding fails (invalid magic, version, or malformed data)

Example:

val, err := cowrie.Decode(data)
if err != nil {
    log.Fatal(err)
}

if val.Type() == cowrie.TypeObject {
    name := val.Get("name").String()
    fmt.Println("Name:", name)
}

DecodeWithOptions

func DecodeWithOptions(data []byte, opts DecodeOptions) (*Value, error)

Decodes with custom security limits and extension handling.

data

[]byte

required

Encoded binary data

opts

DecodeOptions

required

Security limits and options

DecodeOptions fields:

MaxDepth

int

default:"1000"

Maximum nesting depth for arrays/objects

MaxArrayLen

int

default:"100000000"

Maximum array element count (100M)

MaxObjectLen

int

default:"10000000"

Maximum object field count (10M)

MaxStringLen

int

default:"500000000"

Maximum string byte length (500MB)

MaxBytesLen

int

default:"1000000000"

Maximum bytes length (1GB, also applies to tensor/image/audio data)

MaxExtLen

int

default:"100000000"

Maximum TagExt payload length (100MB)

MaxDictLen

int

default:"10000000"

Maximum dictionary entries (10M)

MaxHintCount

int

default:"10000"

Maximum column hints (10K)

MaxRank

int

default:"32"

Maximum tensor rank/dimensions

OnUnknownExt

UnknownExtBehavior

default:"UnknownExtKeep"

Behavior for unknown TagExt extensions:

UnknownExtKeep: Preserve as UnknownExtData (default, enables round-tripping)
UnknownExtSkipAsNull: Skip and return Null
UnknownExtError: Return error in strict mode

Example:

opts := cowrie.DecodeOptions{
    MaxDepth: 500,
    MaxArrayLen: 1000000,
    OnUnknownExt: cowrie.UnknownExtSkipAsNull,
}
val, err := cowrie.DecodeWithOptions(data, opts)

DecodeFrom

func DecodeFrom(rd io.Reader) (*Value, error)

Decodes from an io.Reader. Reads entire input into memory.

For untrusted input, use DecodeFromLimited to prevent OOM attacks.

Example:

file, _ := os.Open("data.cowrie")
defer file.Close()
val, err := cowrie.DecodeFrom(file)

DecodeFromLimited

func DecodeFromLimited(rd io.Reader, maxBytes int64) (*Value, error)

Decodes from an io.Reader with a size limit. Returns ErrInputTooLarge if input exceeds maxBytes.

io.Reader

required

Reader to decode from

maxBytes

int64

required

Maximum input size in bytes

Example:

// Limit to 10MB
val, err := cowrie.DecodeFromLimited(resp.Body, 10*1024*1024)
if err == cowrie.ErrInputTooLarge {
    return fmt.Errorf("response too large")
}

DecodeWithHints

func DecodeWithHints(data []byte) (*HintResult, error)

Decodes data and returns both the value and any embedded column hints.

HintResult.Value

*Value

Decoded value

HintResult.Hints

[]ColumnHint

Column hints (nil if none present)

Example:

result, err := cowrie.DecodeWithHints(data)
if err != nil {
    log.Fatal(err)
}

for _, hint := range result.Hints {
    fmt.Printf("Field %s: type=%s\n", hint.Field, hint.Type)
}

Compression Functions

EncodeFramed

func EncodeFramed(v *Value, comp Compression) ([]byte, error)

Encodes with optional compression. Automatically skips compression if payload is too small or compression doesn’t help.

*Value

required

Value to encode

comp

Compression

required

Compression type:

CompressionNone (0)
CompressionGzip (1)
CompressionZstd (2)

Compression flags (byte 3 of header):

Bit 0: FlagCompressed (0x01) - data is compressed
Bits 1-2: Compression type (0=none, 1=gzip, 2=zstd)
Bit 3: FlagHasColumnHints (0x08) - column hints present

Framing format (when compressed):

Header (4 bytes): Magic + Version + Flags
OrigLen: uvarint (uncompressed size)
CompressedPayload: bytes

Example:

// Encode with zstd compression
data, err := cowrie.EncodeFramed(largeValue, cowrie.CompressionZstd)

// Small payloads may skip compression automatically
if data[3] & cowrie.FlagCompressed == 0 {
    fmt.Println("Compression skipped (not beneficial)")
}

DecodeFramed

func DecodeFramed(data []byte) (*Value, error)

Decodes with automatic decompression. Uses default 1GB decompressed size limit.

data

[]byte

required

Encoded data (compressed or uncompressed)

Example:

// Automatically detects and decompresses if needed
val, err := cowrie.DecodeFramed(data)

DecodeFramedWithLimit

func DecodeFramedWithLimit(data []byte, maxDecompressedSize int) (*Value, error)

Decodes with custom decompressed size limit. Returns ErrDecompressedTooLarge if claimed decompressed size exceeds limit.

data

[]byte

required

Encoded data

maxDecompressedSize

int

required

Maximum decompressed size (prevents decompression bombs)

Example:

// Limit decompressed size to 100MB
val, err := cowrie.DecodeFramedWithLimit(data, 100*1024*1024)
if err == cowrie.ErrDecompressedTooLarge {
    return fmt.Errorf("decompression bomb detected")
}

Error Types

Decoding errors:

ErrInvalidMagic - Magic bytes != ‘SJ’
ErrInvalidVersion - Version != 0x02
ErrUnexpectedEOF - Truncated data
ErrInvalidTag - Unknown type tag
ErrInvalidFieldID - Field ID >= dictionary length
ErrInvalidVarint - Malformed or overflow varint

Security limit errors:

ErrMalformedLength - Length exceeds remaining data
ErrDepthExceeded - Nesting depth > MaxDepth
ErrArrayTooLarge - Array length > MaxArrayLen
ErrObjectTooLarge - Object fields > MaxObjectLen
ErrStringTooLarge - String length > MaxStringLen
ErrBytesTooLarge - Bytes length > MaxBytesLen
ErrExtTooLarge - Extension payload > MaxExtLen
ErrDictTooLarge - Dictionary size > MaxDictLen
ErrInputTooLarge - Input exceeds size limit
ErrDecompressedTooLarge - Decompressed size exceeds limit
ErrUnknownExt - Unknown extension in strict mode

Type-specific errors:

ErrInvalidDType - Invalid tensor dtype
ErrInvalidImgFormat - Invalid image format
ErrInvalidAudioEnc - Invalid audio encoding
ErrInvalidIDWidth - Invalid adjacency list ID width
ErrInvalidDeltaOp - Invalid delta opcode

Dictionary Coding

Gen2 uses dictionary coding for object keys to reduce payload size:

Encoding: All unique keys are collected and stored in the header dictionary
Objects: Reference keys by dictionary index (varint) instead of inline strings
Graph types: Node/Edge property keys also use the shared dictionary

Example wire format:

{"name": "Alice", "age": 30}
{"name": "Bob", "age": 25}

Encoded as:

Dictionary: ["name", "age"]
Object 1: {0: "Alice", 1: 30}
Object 2: {0: "Bob", 1: 25}

Size savings: 70-80% reduction for repeated property schemas in arrays of objects.

Best Practices

Performance

Use EncodeAppend with pre-allocated buffers to avoid allocations
Enable Deterministic mode only when needed (adds sorting overhead)
Use compression for payloads > 256 bytes (automatic threshold)
Prefer zstd over gzip for better compression ratios

Security

Always use DecodeFromLimited for untrusted network input
Set appropriate MaxBytesLen for tensor/image data in your domain
Use UnknownExtError mode if forward compatibility isn’t needed
Validate decompressed size limits to prevent decompression bombs

Compatibility

Use OnUnknownExt = UnknownExtKeep (default) for round-tripping
Preserve unknown extensions when proxying/forwarding data
Check magic bytes ‘SJ’ to distinguish Gen2 from Gen1 format

Gen1 API

Gen2 API

Graph API

ML API

Encode & Decode

Overview

Encoding Functions

Encode

EncodeAppend

EncodeTo

EncodeWithOptions

EncodeWithHints

Decoding Functions

Decode

DecodeWithOptions

DecodeFrom

DecodeFromLimited

DecodeWithHints

Compression Functions

EncodeFramed

DecodeFramed

DecodeFramedWithLimit

Error Types

Dictionary Coding

Best Practices

Performance

Security

Compatibility

See Also

Build docs developers (and LLMs) love

Gen1 API

Gen2 API

Graph API

ML API

Documentation Index

​Overview

​Encoding Functions

​Encode

​EncodeAppend

​EncodeTo

​EncodeWithOptions

​EncodeWithHints

​Decoding Functions

​Decode

​DecodeWithOptions

​DecodeFrom

​DecodeFromLimited

​DecodeWithHints

​Compression Functions

​EncodeFramed

​DecodeFramed

​DecodeFramedWithLimit

​Error Types

​Dictionary Coding

​Best Practices

​Performance

​Security

​Compatibility

​See Also

Build docs developers (and LLMs) love

Overview

Encoding Functions

Encode

EncodeAppend

EncodeTo

EncodeWithOptions

EncodeWithHints

Decoding Functions

Decode

DecodeWithOptions

DecodeFrom

DecodeFromLimited

DecodeWithHints

Compression Functions

EncodeFramed

DecodeFramed

DecodeFramedWithLimit

Error Types

Dictionary Coding

Best Practices

Performance

Security

Compatibility

See Also