Statistics and Monitoring

Overview

RocksDB provides comprehensive statistics for monitoring database performance, diagnosing issues, and tuning configuration. The Statistics object collects counters and histograms across all database operations.

Enabling statistics is essential for production deployments to understand performance characteristics and troubleshoot issues.

Creating Statistics

From statistics.h:859-860:

#include "rocksdb/statistics.h"

std::shared_ptr<Statistics> CreateDBStatistics();

Basic setup:

Options options;
options.statistics = CreateDBStatistics();

DB* db;
Status s = DB::Open(options, "/tmp/testdb", &db);

// Access statistics
uint64_t cache_hits = options.statistics->getTickerCount(BLOCK_CACHE_HIT);

Statistics Levels

From statistics.h:752-775, control overhead vs. detail:

enum StatsLevel : uint8_t {
  kDisableAll,              // No stats
  kExceptTickers = kDisableAll,
  kExceptHistogramOrTimers, // Disable timer stats and histograms
  kExceptTimers,            // Skip timer stats
  kExceptDetailedTimers,    // Skip detailed timers (default)
  kExceptTimeForMutex,      // Skip mutex timing
  kAll,                     // All stats including mutex timing
};

Setting the level:

auto stats = CreateDBStatistics();
stats->set_stats_level(StatsLevel::kExceptDetailedTimers);  // Default

// For production with minimal overhead
stats->set_stats_level(StatsLevel::kExceptTimers);

// For detailed debugging
stats->set_stats_level(StatsLevel::kAll);

kAll includes mutex wait time measurement which can reduce scalability under high concurrency.

Tickers (Counters)

From statistics.h:31-587, RocksDB tracks over 100 counters:

Block Cache Statistics

// Cache hits and misses
BLOCK_CACHE_HIT
BLOCK_CACHE_MISS
BLOCK_CACHE_ADD
BLOCK_CACHE_ADD_FAILURES

// By block type
BLOCK_CACHE_INDEX_HIT
BLOCK_CACHE_INDEX_MISS
BLOCK_CACHE_FILTER_HIT
BLOCK_CACHE_FILTER_MISS
BLOCK_CACHE_DATA_HIT
BLOCK_CACHE_DATA_MISS

// Bytes transferred
BLOCK_CACHE_BYTES_READ
BLOCK_CACHE_BYTES_WRITE

Calculate hit rate:

uint64_t hits = stats->getTickerCount(BLOCK_CACHE_HIT);
uint64_t misses = stats->getTickerCount(BLOCK_CACHE_MISS);
uint64_t total = hits + misses;

if (total > 0) {
  double hit_rate = 100.0 * hits / total;
  printf("Block cache hit rate: %.2f%%\n", hit_rate);
}

Bloom Filter Statistics

// Filter avoided disk reads
BLOOM_FILTER_USEFUL

// Filter checks
BLOOM_FILTER_FULL_POSITIVE      // Filter said "might exist"
BLOOM_FILTER_FULL_TRUE_POSITIVE // Key actually existed

// Prefix filter stats
BLOOM_FILTER_PREFIX_CHECKED
BLOOM_FILTER_PREFIX_USEFUL
BLOOM_FILTER_PREFIX_TRUE_POSITIVE

Measure filter effectiveness:

uint64_t useful = stats->getTickerCount(BLOOM_FILTER_USEFUL);
uint64_t positive = stats->getTickerCount(BLOOM_FILTER_FULL_POSITIVE);
uint64_t total_checks = useful + positive;

if (total_checks > 0) {
  double effectiveness = 100.0 * useful / total_checks;
  printf("Filter effectiveness: %.2f%%\n", effectiveness);
}

Read/Write Statistics

// Operations
NUMBER_KEYS_WRITTEN
NUMBER_KEYS_READ
NUMBER_KEYS_UPDATED

// Bytes
BYTES_WRITTEN
BYTES_READ

// Seeks and iterations
NUMBER_DB_SEEK
NUMBER_DB_NEXT
NUMBER_DB_PREV
NUMBER_DB_SEEK_FOUND
NUMBER_DB_NEXT_FOUND
NUMBER_DB_PREV_FOUND

// MultiGet
NUMBER_MULTIGET_CALLS
NUMBER_MULTIGET_KEYS_READ
NUMBER_MULTIGET_BYTES_READ
NUMBER_MULTIGET_KEYS_FOUND

Compaction Statistics

// Compaction bytes
COMPACT_READ_BYTES
COMPACT_WRITE_BYTES
FLUSH_WRITE_BYTES

// By compaction reason
COMPACT_READ_BYTES_MARKED
COMPACT_READ_BYTES_PERIODIC
COMPACT_READ_BYTES_TTL

// Key drops during compaction
COMPACTION_KEY_DROP_NEWER_ENTRY
COMPACTION_KEY_DROP_OBSOLETE
COMPACTION_KEY_DROP_RANGE_DEL
COMPACTION_KEY_DROP_USER

// Compaction events
COMPACTION_CANCELLED
COMPACTION_ABORTED

Compression Statistics

From statistics.h:251-283:

NUMBER_BLOCK_COMPRESSED
NUMBER_BLOCK_DECOMPRESSED

BYTES_COMPRESSED_FROM      // Uncompressed input
BYTES_COMPRESSED_TO        // Compressed output
BYTES_COMPRESSION_BYPASSED // Stored uncompressed
BYTES_COMPRESSION_REJECTED // Failed compression threshold

BYTES_DECOMPRESSED_FROM    // Compressed input
BYTES_DECOMPRESSED_TO      // Uncompressed output

Stall Statistics

STALL_MICROS  // Total time writers stalled

// Write delays by reason
DB_MUTEX_WAIT_MICROS

Histograms

From statistics.h:604-733, track latency distributions:

enum Histograms : uint32_t {
  DB_GET,                    // Get() latency
  DB_WRITE,                  // Write() latency
  DB_MULTIGET,               // MultiGet() latency
  DB_SEEK,                   // Seek() latency
  
  COMPACTION_TIME,           // Compaction duration
  COMPACTION_CPU_TIME,       // CPU time in compaction
  
  READ_BLOCK_COMPACTION_MICROS,
  READ_BLOCK_GET_MICROS,
  WRITE_RAW_BLOCK_MICROS,
  
  SST_READ_MICROS,           // SST read time
  SST_WRITE_MICROS,          // SST write time
  
  BYTES_PER_READ,            // Value size distribution
  BYTES_PER_WRITE,
  BYTES_PER_MULTIGET,
  
  COMPRESSION_TIMES_NANOS,
  DECOMPRESSION_TIMES_NANOS,
  
  FLUSH_TIME,                // Flush duration
};

Reading Histogram Data

From statistics.h:738-750:

struct HistogramData {
  double median;
  double percentile95;
  double percentile99;
  double average;
  double standard_deviation;
  double max;
  uint64_t count;
  uint64_t sum;
  double min;
};

Example usage:

HistogramData hist;
stats->histogramData(DB_GET, &hist);

printf("Get Latency:\n");
printf("  Median: %.2f us\n", hist.median);
printf("  P95:    %.2f us\n", hist.percentile95);
printf("  P99:    %.2f us\n", hist.percentile99);
printf("  Avg:    %.2f us\n", hist.average);
printf("  Max:    %.2f us\n", hist.max);
printf("  Count:  %lu\n", hist.count);

Accessing Statistics

Get Individual Ticker

From statistics.h:801-806:

uint64_t value = stats->getTickerCount(BLOCK_CACHE_HIT);

Reset Statistics

From statistics.h:829-830:

Status s = stats->Reset();

Get All Tickers as Map

From statistics.h:839-842:

std::map<std::string, uint64_t> ticker_map;
if (stats->getTickerMap(&ticker_map)) {
  for (const auto& [name, value] : ticker_map) {
    printf("%s: %lu\n", name.c_str(), value);
  }
}

String Representation

From statistics.h:833-837:

std::string stats_str = stats->ToString();
printf("%s\n", stats_str.c_str());

Database Properties

Beyond Statistics, query runtime properties:

// Numeric properties
uint64_t num_files_at_level;
db->GetIntProperty("rocksdb.num-files-at-level0", &num_files_at_level);

// String properties
std::string stats;
db->GetProperty("rocksdb.stats", &stats);
printf("%s\n", stats.c_str());

// Map properties
std::map<std::string, uint64_t> values;
db->GetMapProperty(DB::Properties::kBlockCacheEntryStats, &values);

Common Properties

// LSM structure
"rocksdb.num-files-at-level<N>"  // Files at level N
"rocksdb.num-files-at-levels"    // Files at all levels
"rocksdb.total-sst-files-size"   // Total SST size

// Memtable
"rocksdb.num-immutable-mem-table"
"rocksdb.mem-table-flush-pending"
"rocksdb.cur-size-active-mem-table"
"rocksdb.cur-size-all-mem-tables"

// Compaction
"rocksdb.compaction-pending"
"rocksdb.background-errors"
"rocksdb.estimate-pending-compaction-bytes"

// Block cache (map property)
DB::Properties::kBlockCacheEntryStats

Monitoring Patterns

Periodic Statistics Dump

Options options;
options.statistics = CreateDBStatistics();
options.stats_dump_period_sec = 600;  // Dump every 10 minutes

Custom Monitoring Loop

void MonitorDatabase(DB* db, std::shared_ptr<Statistics> stats) {
  while (running) {
    // Cache statistics
    uint64_t cache_hit = stats->getTickerCount(BLOCK_CACHE_HIT);
    uint64_t cache_miss = stats->getTickerCount(BLOCK_CACHE_MISS);
    double hit_rate = 100.0 * cache_hit / (cache_hit + cache_miss);
    
    // Write statistics
    uint64_t bytes_written = stats->getTickerCount(BYTES_WRITTEN);
    uint64_t keys_written = stats->getTickerCount(NUMBER_KEYS_WRITTEN);
    
    // Compaction statistics
    uint64_t compact_read = stats->getTickerCount(COMPACT_READ_BYTES);
    uint64_t compact_write = stats->getTickerCount(COMPACT_WRITE_BYTES);
    double write_amp = static_cast<double>(compact_write) / bytes_written;
    
    // Stall statistics
    uint64_t stall_micros = stats->getTickerCount(STALL_MICROS);
    
    // Log metrics
    LOG(INFO) << "Cache hit rate: " << hit_rate << "%";
    LOG(INFO) << "Write amplification: " << write_amp;
    LOG(INFO) << "Stall time: " << stall_micros / 1000000.0 << " sec";
    
    std::this_thread::sleep_for(std::chrono::seconds(60));
  }
}

Latency Monitoring

void MonitorLatency(std::shared_ptr<Statistics> stats) {
  HistogramData get_hist, write_hist;
  
  stats->histogramData(DB_GET, &get_hist);
  stats->histogramData(DB_WRITE, &write_hist);
  
  printf("Get Latency - P50: %.2f us, P95: %.2f us, P99: %.2f us\n",
         get_hist.median, get_hist.percentile95, get_hist.percentile99);
  
  printf("Write Latency - P50: %.2f us, P95: %.2f us, P99: %.2f us\n",
         write_hist.median, write_hist.percentile95, write_hist.percentile99);
  
  // Alert on high latency
  if (get_hist.percentile99 > 10000) {  // 10ms
    LOG(WARNING) << "High Get P99 latency: " << get_hist.percentile99 << " us";
  }
}

Write Amplification Tracking

double CalculateWriteAmplification(std::shared_ptr<Statistics> stats) {
  uint64_t bytes_written = stats->getTickerCount(BYTES_WRITTEN);
  uint64_t compact_write = stats->getTickerCount(COMPACT_WRITE_BYTES);
  uint64_t flush_write = stats->getTickerCount(FLUSH_WRITE_BYTES);
  
  if (bytes_written == 0) return 0.0;
  
  uint64_t total_write = compact_write + flush_write;
  return static_cast<double>(total_write) / bytes_written;
}

Performance Overhead

Minimal Overhead Configuration

auto stats = CreateDBStatistics();
stats->set_stats_level(StatsLevel::kExceptTimers);

Options options;
options.statistics = stats;

Maximum Detail (Debug)

auto stats = CreateDBStatistics();
stats->set_stats_level(StatsLevel::kAll);

Options options;
options.statistics = stats;

Most tickers have negligible overhead. Histograms and detailed timers add measurable but usually acceptable overhead.

Integration with Monitoring Systems

Prometheus Exporter

class PrometheusExporter {
 public:
  void Export(std::shared_ptr<Statistics> stats) {
    // Export tickers
    for (uint32_t i = 0; i < TICKER_ENUM_MAX; i++) {
      uint64_t value = stats->getTickerCount(i);
      const auto& [ticker, name] = TickersNameMap[i];
      prometheus_gauge_.Set(name, value);
    }
    
    // Export histogram percentiles
    HistogramData hist;
    stats->histogramData(DB_GET, &hist);
    prometheus_histogram_.Observe("db_get_p50", hist.median);
    prometheus_histogram_.Observe("db_get_p95", hist.percentile95);
    prometheus_histogram_.Observe("db_get_p99", hist.percentile99);
  }
};

Troubleshooting with Statistics

High Write Amplification

Check:

uint64_t bytes_written = stats->getTickerCount(BYTES_WRITTEN);
uint64_t compact_write = stats->getTickerCount(COMPACT_WRITE_BYTES);
double write_amp = static_cast<double>(compact_write) / bytes_written;

if (write_amp > 10.0) {
  // Investigate compaction configuration
  // Consider level_compaction_dynamic_level_bytes
  // Check num_levels and target_file_size_base
}

Cache Inefficiency

Check:

uint64_t hits = stats->getTickerCount(BLOCK_CACHE_HIT);
uint64_t misses = stats->getTickerCount(BLOCK_CACHE_MISS);
double hit_rate = 100.0 * hits / (hits + misses);

if (hit_rate < 80.0) {
  // Increase cache size
  // Pin important blocks (index/filter)
  // Check working set size vs cache capacity
}

Write Stalls

Check:

uint64_t stall_micros = stats->getTickerCount(STALL_MICROS);

if (stall_micros > 0) {
  // Check level0_slowdown_writes_trigger
  // Check soft_pending_compaction_bytes_limit
  // Increase write_buffer_size
  // Add more background threads
}

Best Practices

Always enable statistics in production with at least kExceptDetailedTimers level.

Use appropriate stats level: Balance overhead and detail
Monitor key metrics: Hit rate, write amp, latency, stalls
Set up alerting: Alert on anomalies (high P99, low hit rate)
Periodic dumps: Use stats_dump_period_sec for logging
Reset when needed: Call Reset() after major events
Correlate with workload: Understand metrics in context of operations

​Overview

​Creating Statistics

​Statistics Levels

​Tickers (Counters)

​Block Cache Statistics

​Bloom Filter Statistics

​Read/Write Statistics

​Compaction Statistics

​Compression Statistics

​Stall Statistics

​Histograms

​Reading Histogram Data

​Accessing Statistics

​Get Individual Ticker

​Reset Statistics

​Get All Tickers as Map

​String Representation

​Database Properties

​Common Properties

​Monitoring Patterns

​Periodic Statistics Dump

​Custom Monitoring Loop

​Latency Monitoring

​Write Amplification Tracking

​Performance Overhead

​Minimal Overhead Configuration

​Maximum Detail (Debug)

​Integration with Monitoring Systems

​Prometheus Exporter

​Troubleshooting with Statistics

​High Write Amplification

​Cache Inefficiency

​Write Stalls

​Best Practices

​See Also

Overview

Creating Statistics

Statistics Levels

Tickers (Counters)

Block Cache Statistics

Bloom Filter Statistics

Read/Write Statistics

Compaction Statistics

Compression Statistics

Stall Statistics

Histograms

Reading Histogram Data

Accessing Statistics

Get Individual Ticker

Reset Statistics

Get All Tickers as Map

String Representation

Database Properties

Common Properties

Monitoring Patterns

Periodic Statistics Dump

Custom Monitoring Loop

Latency Monitoring

Write Amplification Tracking

Performance Overhead

Minimal Overhead Configuration

Maximum Detail (Debug)

Integration with Monitoring Systems

Prometheus Exporter

Troubleshooting with Statistics

High Write Amplification

Cache Inefficiency

Write Stalls

Best Practices

See Also