BackupEngine

BackupEngine provides APIs for creating and managing consistent backups of RocksDB databases. It supports incremental backups, backup verification, and flexible restore options.

Opening a BackupEngine

Open

static IOStatus Open(
  const BackupEngineOptions& options,
  Env* db_env,
  BackupEngine** backup_engine_ptr
);

Opens a BackupEngine with the specified options.

const BackupEngineOptions&

Configuration for the backup engine including backup directory and sharing options.

Env*

Environment used by the database being backed up.

BackupEngine**

Output parameter for the opened BackupEngine.

IOStatus

Returns OK on success.

Creating Backups

CreateNewBackup

virtual IOStatus CreateNewBackup(
  const CreateBackupOptions& options,
  DB* db,
  BackupID* new_backup_id = nullptr
);

Captures the state of the database by creating a new backup.

const CreateBackupOptions&

Options including whether to flush before backup and progress callback.

DB*

Database to back up.

BackupID*

Optional output parameter for the new backup’s ID.

IOStatus

Returns OK on success.

CreateNewBackupWithMetadata

virtual IOStatus CreateNewBackupWithMetadata(
  const CreateBackupOptions& options,
  DB* db,
  const std::string& app_metadata,
  BackupID* new_backup_id = nullptr
);

Creates a backup with application-specific metadata.

Backup Management

GetBackupInfo

virtual void GetBackupInfo(
  std::vector<BackupInfo>* backup_infos,
  bool include_file_details = false
) const;

Retrieves information about all non-corrupt backups.

std::vector<BackupInfo>*

Output vector of backup information.

bool

If true, includes detailed file information for each backup.

GetLatestBackupInfo

virtual Status GetLatestBackupInfo(
  BackupInfo* backup_info,
  bool include_file_details = false
) const;

Retrieves information about the latest good backup.

DeleteBackup

virtual IOStatus DeleteBackup(BackupID backup_id);

Deletes a specific backup.

BackupID

ID of the backup to delete.

PurgeOldBackups

virtual IOStatus PurgeOldBackups(uint32_t num_backups_to_keep);

Deletes old backups, keeping only the specified number of latest backups.

GarbageCollect

virtual IOStatus GarbageCollect();

Deletes files left over from incomplete backup operations.

Restoring Backups

RestoreDBFromBackup

virtual IOStatus RestoreDBFromBackup(
  const RestoreOptions& options,
  BackupID backup_id,
  const std::string& db_dir,
  const std::string& wal_dir
) const;

Restores a database from a specific backup.

const RestoreOptions&

Restore options including incremental restore mode.

BackupID

ID of the backup to restore.

const std::string&

Target directory for database files.

const std::string&

Target directory for WAL files.

IOStatus

Returns OK on success.

RestoreDBFromLatestBackup

virtual IOStatus RestoreDBFromLatestBackup(
  const RestoreOptions& options,
  const std::string& db_dir,
  const std::string& wal_dir
) const;

Restores from the latest non-corrupt backup.

Backup Verification

VerifyBackup

virtual IOStatus VerifyBackup(
  BackupID backup_id,
  bool verify_with_checksum = false
) const;

Verifies a backup’s integrity.

BackupID

ID of the backup to verify.

bool

If true, verifies file checksums. If false, only checks file existence and size.

Configuration

BackupEngineOptions

struct BackupEngineOptions {
  std::string backup_dir;
  Env* backup_env = nullptr;
  bool share_table_files = true;
  Logger* info_log = nullptr;
  bool sync = true;
  bool destroy_old_data = false;
  bool backup_log_files = true;
  uint64_t io_buffer_size = 0;
  uint64_t backup_rate_limit = 0;
  std::shared_ptr<RateLimiter> backup_rate_limiter = nullptr;
  uint64_t restore_rate_limit = 0;
  std::shared_ptr<RateLimiter> restore_rate_limiter = nullptr;
  bool share_files_with_checksum = true;
  int max_background_operations = 1;
  uint64_t callback_trigger_interval_size = 4194304;
  ShareFilesNaming share_files_with_checksum_naming;
  int schema_version = 1;
};

std::string

Directory where backups are stored. Required.

If true, table and blob files are shared among backups for space savings and incremental backups.

bool

If true, guarantees consistent backup/restore even on machine crash.

uint64_t

Max bytes per second during backup. 0 means unlimited.

int

Number of threads for parallel backup/restore operations.

CreateBackupOptions

struct CreateBackupOptions {
  bool flush_before_backup = false;
  std::function<void()> progress_callback = {};
  std::function<void(MaybeExcludeBackupFile* files_begin,
                     MaybeExcludeBackupFile* files_end)>
      exclude_files_callback = {};
  bool decrease_background_thread_cpu_priority = false;
  CpuPriority background_thread_cpu_priority = CpuPriority::kNormal;
};

bool

Whether to flush memtable before backup. Always triggered if 2PC is enabled.

std::function<void()>

Callback for progress reporting based on callback_trigger_interval_size.

std::function

Advanced callback to exclude shared files from backup.

RestoreOptions

struct RestoreOptions {
  enum Mode : uint32_t {
    kKeepLatestDbSessionIdFiles = 1U,
    kVerifyChecksum = 2U,
    kPurgeAllFiles = 0xffffU
  };
  bool keep_log_files = false;
  std::forward_list<BackupEngineReadOnlyBase*> alternate_dirs;
  Mode mode = Mode::kPurgeAllFiles;
};

Mode

Restore strategy: kKeepLatestDbSessionIdFiles (incremental), kVerifyChecksum (verify then restore), or kPurgeAllFiles (full restore).

bool

If true, don’t overwrite existing log files.

BackupInfo Structure

struct BackupInfo {
  BackupID backup_id;
  int64_t timestamp;
  uint64_t size;
  uint32_t number_files;
  std::string app_metadata;
  std::vector<BackupFileInfo> file_details;
  std::vector<BackupExcludedFileInfo> excluded_files;
  std::string name_for_open;
  std::shared_ptr<Env> env_for_open;
};

BackupID

Unique identifier for the backup.

int64_t

Creation time from GetCurrentTime.

uint64_t

Total size in bytes.

std::string

Application-specific metadata.

Example: Creating and Restoring Backups

#include "rocksdb/utilities/backup_engine.h"

using namespace ROCKSDB_NAMESPACE;

// Open database
DB* db;
Options options;
options.create_if_missing = true;
Status s = DB::Open(options, "/tmp/testdb", &db);

// Create backup engine
BackupEngine* backup_engine;
BackupEngineOptions backup_options("/tmp/backups");
backup_options.share_table_files = true;
backup_options.max_background_operations = 4;

IOStatus io_s = BackupEngine::Open(backup_options, Env::Default(),
                                    &backup_engine);
assert(io_s.ok());

// Write some data
db->Put(WriteOptions(), "key1", "value1");

// Create backup
CreateBackupOptions create_opts;
create_opts.flush_before_backup = false;
BackupID backup_id;
io_s = backup_engine->CreateNewBackup(create_opts, db, &backup_id);
assert(io_s.ok());

// Get backup info
std::vector<BackupInfo> backup_info;
backup_engine->GetBackupInfo(&backup_info);
for (const auto& info : backup_info) {
  printf("Backup %u: %lu bytes, %u files\n",
         info.backup_id, info.size, info.number_files);
}

// Verify backup
io_s = backup_engine->VerifyBackup(backup_id, true);
assert(io_s.ok());

// Close and restore
delete db;

RestoreOptions restore_opts;
io_s = backup_engine->RestoreDBFromBackup(restore_opts, backup_id,
                                           "/tmp/testdb", "/tmp/testdb");
assert(io_s.ok());

// Reopen restored database
s = DB::Open(options, "/tmp/testdb", &db);
assert(s.ok());

delete backup_engine;
delete db;

ShareFilesNaming

enum ShareFilesNaming : uint32_t {
  kLegacyCrc32cAndFileSize = 1U,
  kUseDbSessionId = 2U,
  kFlagIncludeFileSize = 1U << 31
};

ShareFilesNaming

Recommended. Uses file_number and db_session_id for unique file naming.

ShareFilesNaming

Legacy naming using CRC32c and file size. Only for preserving old behavior.

Stopping Backups

StopBackup

virtual void StopBackup();

Stops an in-progress backup. The backup will stop ASAP and return Status::Incomplete(). This is a one-way operation - to create new backups, open a new BackupEngine.

Database

Read & Write

Transactions

Storage

Utilities

Opening a BackupEngine

Open

Creating Backups

CreateNewBackup

CreateNewBackupWithMetadata

Backup Management

GetBackupInfo

GetLatestBackupInfo

DeleteBackup

PurgeOldBackups

GarbageCollect

Restoring Backups

RestoreDBFromBackup

RestoreDBFromLatestBackup

Backup Verification

VerifyBackup

Configuration

BackupEngineOptions

CreateBackupOptions

RestoreOptions

BackupInfo Structure

Example: Creating and Restoring Backups

ShareFilesNaming

Stopping Backups

StopBackup

See Also

​Opening a BackupEngine

​Open

​Creating Backups

​CreateNewBackup

​CreateNewBackupWithMetadata

​Backup Management

​GetBackupInfo

​GetLatestBackupInfo

​DeleteBackup

​PurgeOldBackups

​GarbageCollect

​Restoring Backups

​RestoreDBFromBackup

​RestoreDBFromLatestBackup

​Backup Verification

​VerifyBackup

​Configuration

​BackupEngineOptions

​CreateBackupOptions

​RestoreOptions

​BackupInfo Structure

​Example: Creating and Restoring Backups

​File Sharing and Naming

​ShareFilesNaming

​Stopping Backups

​StopBackup

​See Also

Opening a BackupEngine

Open

Creating Backups

CreateNewBackup

CreateNewBackupWithMetadata

Backup Management

GetBackupInfo

GetLatestBackupInfo

DeleteBackup

PurgeOldBackups

GarbageCollect

Restoring Backups

RestoreDBFromBackup

RestoreDBFromLatestBackup

Backup Verification

VerifyBackup

Configuration

BackupEngineOptions

CreateBackupOptions

RestoreOptions

BackupInfo Structure

Example: Creating and Restoring Backups

File Sharing and Naming

ShareFilesNaming

Stopping Backups

StopBackup

See Also