FileSystem - RocksDB

FileSystem is a modern interface for file system operations that provides more fine-grained control than the Env interface. It supports features like async I/O, temperature-aware storage, and detailed I/O options.

Getting a FileSystem

Default

static std::shared_ptr<FileSystem> Default();

Returns the default FileSystem for the current platform.

std::shared_ptr<FileSystem>

Shared pointer to the default FileSystem.

File Creation

NewSequentialFile

virtual IOStatus NewSequentialFile(
  const std::string& fname,
  const FileOptions& file_opts,
  std::unique_ptr<FSSequentialFile>* result,
  IODebugContext* dbg
);

Creates a sequentially-readable file.

const std::string&

Path to the file.

const FileOptions&

File options including I/O options, temperature, and checksum settings.

std::unique_ptr<FSSequentialFile>*

Output parameter for the file object.

IODebugContext*

Debug context for troubleshooting.

IOStatus

Returns OK on success, with detailed error information on failure.

NewRandomAccessFile

virtual IOStatus NewRandomAccessFile(
  const std::string& fname,
  const FileOptions& file_opts,
  std::unique_ptr<FSRandomAccessFile>* result,
  IODebugContext* dbg
);

Creates a random access read-only file.

NewWritableFile

virtual IOStatus NewWritableFile(
  const std::string& fname,
  const FileOptions& file_opts,
  std::unique_ptr<FSWritableFile>* result,
  IODebugContext* dbg
);

Creates a writable file.

FileOptions

struct FileOptions : EnvOptions {
  IOOptions io_options;
  Temperature temperature = Temperature::kUnknown;
  ChecksumType handoff_checksum_type;
  Env::WriteLifeTimeHint write_hint = Env::WLTH_NOT_SET;
  std::string file_checksum;
  std::string file_checksum_func_name;
};

Temperature

Hint for storage media placement: kUnknown, kHot, kWarm, kCold.

ChecksumType

Checksum type for write verification.

WriteLifeTimeHint

Expected lifetime of the data for storage optimization.

IOOptions

struct IOOptions {
  std::chrono::microseconds timeout;
  IOPriority prio;
  Env::IOPriority rate_limiter_priority;
  IOType type;
  std::unordered_map<std::string, std::string> property_bag;
  bool force_dir_fsync;
  bool do_not_recurse;
  bool verify_and_reconstruct_read;
  Env::IOActivity io_activity = Env::IOActivity::kUnknown;
};

std::chrono::microseconds

Timeout for the I/O operation.

Env::IOPriority

Priority for rate limiting.

IOType

Type of data: kData, kFilter, kIndex, kMetadata, kWAL, kManifest, kLog.

bool

Request stronger data integrity checking and correction on read.

Directory Operations

NewDirectory

virtual IOStatus NewDirectory(
  const std::string& name,
  const IOOptions& io_opts,
  std::unique_ptr<FSDirectory>* result,
  IODebugContext* dbg
);

Creates a directory object for fsync operations.

CreateDir

virtual IOStatus CreateDir(
  const std::string& dirname,
  const IOOptions& options,
  IODebugContext* dbg
);

Creates a directory.

CreateDirIfMissing

virtual IOStatus CreateDirIfMissing(
  const std::string& dirname,
  const IOOptions& options,
  IODebugContext* dbg
);

Creates a directory if it doesn’t exist.

File Operations

DeleteFile

virtual IOStatus DeleteFile(
  const std::string& fname,
  const IOOptions& options,
  IODebugContext* dbg
);

Deletes a file.

RenameFile

virtual IOStatus RenameFile(
  const std::string& src,
  const std::string& target,
  const IOOptions& options,
  IODebugContext* dbg
);

Renames a file.

GetFileSize

virtual IOStatus GetFileSize(
  const std::string& fname,
  const IOOptions& options,
  uint64_t* file_size,
  IODebugContext* dbg
);

Gets file size.

LinkFile

virtual IOStatus LinkFile(
  const std::string& src,
  const std::string& target,
  const IOOptions& options,
  IODebugContext* dbg
);

Creates a hard link.

File System Query

FileExists

virtual IOStatus FileExists(
  const std::string& fname,
  const IOOptions& options,
  IODebugContext* dbg
);

Checks if a file exists.

GetChildren

virtual IOStatus GetChildren(
  const std::string& dir,
  const IOOptions& options,
  std::vector<std::string>* result,
  IODebugContext* dbg
);

Lists directory contents.

GetFreeSpace

virtual IOStatus GetFreeSpace(
  const std::string& path,
  const IOOptions& options,
  uint64_t* diskfree,
  IODebugContext* dbg
);

Gets free disk space.

IsDirectory

virtual IOStatus IsDirectory(
  const std::string& path,
  const IOOptions& options,
  bool* is_dir,
  IODebugContext* dbg
);

Checks if a path is a directory.

Async I/O Support

SupportedOps

virtual void SupportedOps(int64_t& supported_ops);

Indicates which advanced operations are supported:

kAsyncIO: Async read operations
kFSBuffer: File system allocated buffers
kVerifyAndReconstructRead: Enhanced data integrity
kFSPrefetch: Prefetch operations

Poll

virtual IOStatus Poll(
  std::vector<void*>& io_handles,
  size_t min_completions
);

Polls for completion of async I/O requests.

AbortIO

virtual IOStatus AbortIO(
  std::vector<void*>& io_handles
);

Aborts async I/O requests.

IODebugContext

struct IODebugContext {
  std::string file_path;
  std::map<std::string, uint64_t> counters;
  std::string msg;
  const std::string* request_id = nullptr;
  uint64_t trace_data = 0;
  std::any cost_info;
  std::shared_mutex mutex;
};

std::string

File path filled in by RocksDB.

std::map<std::string, uint64_t>

Performance counters set by the FileSystem.

std::string

Debug message from the FileSystem.

const std::string*

Optional request ID for tracing.

Temperature Awareness

enum class Temperature : uint8_t {
  kUnknown = 0,
  kHot = 1,
  kWarm = 2,
  kCold = 3
};

Temperature hints help file systems place data on appropriate storage media:

kHot: Frequently accessed data, place on fast storage
kWarm: Moderately accessed data
kCold: Rarely accessed data, can use slower storage

Example: Basic File Operations

#include "rocksdb/file_system.h"

using namespace ROCKSDB_NAMESPACE;

// Get default file system
std::shared_ptr<FileSystem> fs = FileSystem::Default();

// Configure file options
FileOptions file_opts;
file_opts.temperature = Temperature::kHot;
file_opts.io_options.rate_limiter_priority = Env::IO_HIGH;

// Create a writable file
IODebugContext dbg;
std::unique_ptr<FSWritableFile> file;
IOStatus s = fs->NewWritableFile("/tmp/test.sst", file_opts, &file, &dbg);

if (!s.ok()) {
  fprintf(stderr, "Error: %s\n", s.ToString().c_str());
  fprintf(stderr, "Debug: %s\n", dbg.ToString().c_str());
}

Example: Async I/O

#include "rocksdb/file_system.h"

using namespace ROCKSDB_NAMESPACE;

// Check if async I/O is supported
std::shared_ptr<FileSystem> fs = FileSystem::Default();
int64_t supported_ops = 0;
fs->SupportedOps(supported_ops);

if (supported_ops & (1 << FSSupportedOps::kAsyncIO)) {
  // Async I/O is supported
  // Use FSRandomAccessFile::ReadAsync()
}

FSRandomAccessFile

Read

virtual IOStatus Read(
  uint64_t offset,
  size_t n,
  const IOOptions& options,
  Slice* result,
  char* scratch,
  IODebugContext* dbg
) const;

Reads data at a specific offset.

MultiRead

virtual IOStatus MultiRead(
  FSReadRequest* reqs,
  size_t num_reqs,
  const IOOptions& options,
  IODebugContext* dbg
);

Reads multiple blocks, potentially in parallel.

ReadAsync

virtual IOStatus ReadAsync(
  FSReadRequest& req,
  const IOOptions& opts,
  std::function<void(FSReadRequest&, void*)> cb,
  void* cb_arg,
  void** io_handle,
  IOHandleDeleter* del_fn,
  IODebugContext* dbg
);

Asynchronously reads data.

FSWritableFile

Append

virtual IOStatus Append(
  const Slice& data,
  const IOOptions& options,
  IODebugContext* dbg
);

Appends data to the file.

Append with Checksum

virtual IOStatus Append(
  const Slice& data,
  const IOOptions& options,
  const DataVerificationInfo& verification_info,
  IODebugContext* dbg
);

Appends data with checksum verification.

Sync

virtual IOStatus Sync(
  const IOOptions& options,
  IODebugContext* dbg
);

Syncs data to storage.

​Getting a FileSystem

​Default

​File Creation

​NewSequentialFile

​NewRandomAccessFile

​NewWritableFile

​FileOptions

​IOOptions

​Directory Operations

​NewDirectory

​CreateDir

​CreateDirIfMissing

​File Operations

​DeleteFile

​RenameFile

​GetFileSize

​LinkFile

​File System Query

​FileExists

​GetChildren

​GetFreeSpace

​IsDirectory

​Async I/O Support

​SupportedOps

​Poll

​AbortIO

​IODebugContext

​Temperature Awareness

​Example: Basic File Operations

​Example: Async I/O

​FSRandomAccessFile

​Read

​MultiRead

​ReadAsync

​FSWritableFile

​Append

​Append with Checksum

​Sync

​See Also