Merge Operators

Merge operators provide an efficient way to implement read-modify-write semantics in RocksDB. Instead of reading a value, modifying it, and writing it back, you can apply merge operands that are combined later.

Why Use Merge Operators?

Performance

Merge operations are faster than read-modify-write cycles, especially under high concurrency.

Concurrency

Multiple merge operations can be applied concurrently without coordination.

Efficiency

Merge operands are smaller than full values and can be batched together.

Atomicity

Merge operations are atomic and consistent with RocksDB’s guarantees.

Merge Operator Types

RocksDB provides two types of merge operators:

AssociativeMergeOperator

For simple associative operations (e.g., numeric addition, string concatenation):

class AssociativeMergeOperator : public MergeOperator {
public:
    virtual bool Merge(const Slice& key,
                      const Slice* existing_value,
                      const Slice& value,
                      std::string* new_value,
                      Logger* logger) const = 0;
};

MergeOperator

For more complex operations with multiple merge operands:

class MergeOperator {
public:
    virtual bool FullMergeV2(const MergeOperationInput& merge_in,
                            MergeOperationOutput* merge_out) const;
    
    virtual bool PartialMerge(const Slice& key,
                             const Slice& left_operand,
                             const Slice& right_operand,
                             std::string* new_value,
                             Logger* logger) const;
};

Basic Usage

Configuring a Merge Operator

#include "rocksdb/db.h"
#include "rocksdb/merge_operator.h"
#include "rocksdb/options.h"

using ROCKSDB_NAMESPACE::DB;
using ROCKSDB_NAMESPACE::Options;
using ROCKSDB_NAMESPACE::MergeOperator;

// Create a merge operator (example: counter)
class CounterMergeOperator : public AssociativeMergeOperator {
public:
    bool Merge(const Slice& key,
              const Slice* existing_value,
              const Slice& value,
              std::string* new_value,
              Logger* logger) const override {
        // Convert to integers
        int64_t existing = 0;
        if (existing_value) {
            existing = std::stoll(existing_value->ToString());
        }
        
        int64_t operand = std::stoll(value.ToString());
        
        // Add them together
        *new_value = std::to_string(existing + operand);
        return true;
    }
    
    const char* Name() const override {
        return "CounterMergeOperator";
    }
};

// Configure database to use the merge operator
Options options;
options.merge_operator.reset(new CounterMergeOperator());

DB* db;
Status s = DB::Open(options, "/path/to/db", &db);

Using Merge Operations

using ROCKSDB_NAMESPACE::WriteOptions;
using ROCKSDB_NAMESPACE::ReadOptions;

// Initialize counter
db->Put(WriteOptions(), "counter", "0");

// Increment counter using merge
db->Merge(WriteOptions(), "counter", "1");
db->Merge(WriteOptions(), "counter", "5");
db->Merge(WriteOptions(), "counter", "3");

// Read the result
std::string value;
db->Get(ReadOptions(), "counter", &value);
assert(value == "9");  // 0 + 1 + 5 + 3 = 9

Merge operations are particularly useful for counters, append-only lists, and any operation where you need to accumulate changes.

Implementing AssociativeMergeOperator

String Append Example

class StringAppendOperator : public AssociativeMergeOperator {
public:
    bool Merge(const Slice& key,
              const Slice* existing_value,
              const Slice& value,
              std::string* new_value,
              Logger* logger) const override {
        // Concatenate strings
        if (existing_value) {
            *new_value = existing_value->ToString();
        } else {
            *new_value = "";
        }
        
        new_value->append(value.data(), value.size());
        return true;
    }
    
    const char* Name() const override {
        return "StringAppendOperator";
    }
};

Usage:

Options options;
options.merge_operator.reset(new StringAppendOperator());

DB* db;
DB::Open(options, "/path/to/db", &db);

db->Merge(WriteOptions(), "log", "Event 1\n");
db->Merge(WriteOptions(), "log", "Event 2\n");
db->Merge(WriteOptions(), "log", "Event 3\n");

std::string value;
db->Get(ReadOptions(), "log", &value);
// value = "Event 1\nEvent 2\nEvent 3\n"

Implementing Full MergeOperator

For more complex scenarios, implement the full MergeOperator interface:

JSON Merge Example

class JsonMergeOperator : public MergeOperator {
public:
    bool FullMergeV2(const MergeOperationInput& merge_in,
                    MergeOperationOutput* merge_out) const override {
        // Start with existing value or empty object
        std::string result = merge_in.existing_value 
            ? merge_in.existing_value->ToString() 
            : "{}";
        
        // Apply each merge operand
        for (const auto& operand : merge_in.operand_list) {
            result = MergeJsonStrings(result, operand.ToString());
        }
        
        merge_out->new_value = result;
        return true;
    }
    
    bool PartialMerge(const Slice& key,
                     const Slice& left_operand,
                     const Slice& right_operand,
                     std::string* new_value,
                     Logger* logger) const override {
        // Optionally combine two merge operands
        *new_value = MergeJsonStrings(left_operand.ToString(),
                                     right_operand.ToString());
        return true;
    }
    
    const char* Name() const override {
        return "JsonMergeOperator";
    }
    
private:
    std::string MergeJsonStrings(const std::string& left,
                                const std::string& right) const {
        // Implementation to merge JSON objects
        // (simplified for illustration)
        return left + right;
    }
};

MergeOperator API Reference

MergeOperationInput

struct MergeOperationInput {
    // The key associated with the merge operation
    const Slice& key;
    
    // The existing value (nullptr if key doesn't exist)
    const Slice* existing_value;
    
    // List of merge operands to apply
    const std::vector<Slice>& operand_list;
    
    // Logger for error messages
    Logger* logger;
};

MergeOperationOutput

struct MergeOperationOutput {
    // The merged result
    std::string& new_value;
    
    // Or, reference to existing operand (optimization)
    Slice& existing_operand;
};

Advanced Features

PartialMerge

Optimize by combining merge operands before full merge:

bool PartialMerge(const Slice& key,
                 const Slice& left_operand,
                 const Slice& right_operand,
                 std::string* new_value,
                 Logger* logger) const override {
    // Combine two operands into one
    int64_t left = std::stoll(left_operand.ToString());
    int64_t right = std::stoll(right_operand.ToString());
    
    *new_value = std::to_string(left + right);
    return true;
}

PartialMerge is called during compaction to reduce the number of merge operands, improving read performance.

Wide Column Support (FullMergeV3)

For wide-column entities:

bool FullMergeV3(const MergeOperationInputV3& merge_in,
                MergeOperationOutputV3* merge_out) const override {
    // Handle wide-column entities
    // Can merge individual columns independently
    return true;
}

Merge in Transactions

Merge operations work with transactions:

#include "rocksdb/utilities/transaction.h"
#include "rocksdb/utilities/transaction_db.h"

using ROCKSDB_NAMESPACE::Transaction;
using ROCKSDB_NAMESPACE::TransactionDB;

TransactionDB* txn_db;
// Open transaction DB with merge operator...

Transaction* txn = txn_db->BeginTransaction(WriteOptions());

// Merge within transaction
txn->Merge("counter", "10");
txn->Merge("counter", "20");

// Commit atomically
Status s = txn->Commit();
assert(s.ok());

delete txn;

Performance Considerations

Keep merge operands small

Smaller operands are more efficient to store and combine.

Implement PartialMerge

Reduce the number of operands during compaction for better read performance.

Use AllowSingleOperand()

If your merge operator can handle a single operand efficiently, override this method to return true.

Handle errors gracefully

Return false from merge methods if the operation cannot be performed, and log the error.

Common Use Cases

Counters
Append-Only Logs
Set Operations
JSON/Document Updates

Implement atomic increment/decrement operations without explicit locking.

Built-in Merge Operators

RocksDB provides some built-in merge operators:

#include "rocksdb/merge_operator.h"

using ROCKSDB_NAMESPACE::MergeOperators;

// String append operator with delimiter
options.merge_operator = MergeOperators::CreateStringAppendOperator(',');

// Uint64 add operator
options.merge_operator = MergeOperators::CreateUInt64AddOperator();

// Max operator
options.merge_operator = MergeOperators::CreateMaxOperator();

Error Handling

bool FullMergeV2(const MergeOperationInput& merge_in,
                MergeOperationOutput* merge_out) const override {
    try {
        // Perform merge...
        merge_out->new_value = result;
        return true;
    } catch (const std::exception& e) {
        // Log error
        if (merge_in.logger) {
            merge_in.logger->Logv("Merge failed: %s", e.what());
        }
        return false;
    }
}

If a merge operation returns false, RocksDB will treat it as a corruption error and the operation will fail.

Best Practices

Keep operations simple

Merge operators should be deterministic and fast.

Test thoroughly

Test with various combinations of Put, Merge, and Delete operations.

Document semantics

Clearly document what your merge operator does and its assumptions.

Version your operator

If you change merge semantics, use a new operator name to avoid compatibility issues.

Handle missing values

Your merge operator must handle the case where there’s no existing value (nullptr).

Next Steps

Basic Operations

Learn fundamental CRUD operations

Transactions

Use merge operators in transactions

Performance Tuning

Optimize merge performance

Get Started

Core Concepts

Developer Guide

Advanced Topics

Language Bindings

Tools & Utilities

Merge Operators

Why Use Merge Operators?

Performance

Concurrency

Efficiency

Atomicity

Merge Operator Types

AssociativeMergeOperator

MergeOperator

Basic Usage

Configuring a Merge Operator

Using Merge Operations

Implementing AssociativeMergeOperator

String Append Example

Implementing Full MergeOperator

JSON Merge Example

MergeOperator API Reference

MergeOperationInput

MergeOperationOutput

Advanced Features

PartialMerge

Wide Column Support (FullMergeV3)

Merge in Transactions

Performance Considerations

Common Use Cases

Built-in Merge Operators

Error Handling

Best Practices

Next Steps

Basic Operations

Transactions

Performance Tuning

​Why Use Merge Operators?

Performance

Concurrency

Efficiency

Atomicity

​Merge Operator Types

​AssociativeMergeOperator

​MergeOperator

​Basic Usage

​Configuring a Merge Operator

​Using Merge Operations

​Implementing AssociativeMergeOperator

​String Append Example

​Implementing Full MergeOperator

​JSON Merge Example

​MergeOperator API Reference

​MergeOperationInput

​MergeOperationOutput

​Advanced Features

​PartialMerge

​Wide Column Support (FullMergeV3)

​Merge in Transactions

​Performance Considerations

​Common Use Cases

​Built-in Merge Operators

​Error Handling

​Best Practices

​Next Steps

Basic Operations

Transactions

Performance Tuning

Why Use Merge Operators?

Merge Operator Types

AssociativeMergeOperator

MergeOperator

Basic Usage

Configuring a Merge Operator

Using Merge Operations

Implementing AssociativeMergeOperator

String Append Example

Implementing Full MergeOperator

JSON Merge Example

MergeOperator API Reference

MergeOperationInput

MergeOperationOutput

Advanced Features

PartialMerge

Wide Column Support (FullMergeV3)

Merge in Transactions

Performance Considerations

Common Use Cases

Built-in Merge Operators

Error Handling

Best Practices

Next Steps