Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/rpng/open_vins/llms.txt

Use this file to discover all available pages before exploring further.

The ov_eval package is the dedicated evaluation suite bundled with OpenVINS. It provides C++ binaries and Python scripts for computing standard visual-inertial odometry (VIO) accuracy metrics, analyzing estimator consistency, and profiling the computational load of every system component. Fair comparison between methods—and insight into why one method performs better than another—requires a principled evaluation framework, which is exactly what ov_eval is designed to supply.

​
Key metrics

ov_eval targets six categories of evaluation signal:
MetricWhat it measures
Absolute Trajectory Error (ATE)Global accuracy after optimal alignment of estimated and ground-truth trajectories
Relative Pose Error (RPE)Local drift over fixed-length segments; less sensitive to isolated jumps
Root Mean Squared Error (RMSE)Per-timestep position/orientation error across multiple runs
Normalized Estimation Error Squared (NEES)Estimator consistency relative to its own reported covariance
Component timingWall-clock time consumed by each processing stage (tracking, propagation, update, …)
Hardware usageCPU and memory consumption measured via psutil
Detailed mathematical definitions of the accuracy metrics are on the Metrics page. Timing analysis is covered on the Timing page.

​
Building ov_eval

ov_eval is compiled alongside the rest of OpenVINS by default. No extra CMake flag is needed. 
# Standard catkin / colcon build — ov_eval is included automatically
catkin build

# To disable ov_eval (not recommended for research workflows)
catkin build -DBUILD_OV_EVAL=OFF
Plotting inside the C++ binaries requires python-matplotlib, python-numpy, and the Python development headers. Install them with:
sudo apt-get install python-matplotlib python-numpy python2.7-dev
If these are not present, the binaries still run and print results to the console — only the inline figures are skipped.

​
Available evaluation binaries

Every binary below is built into ov_eval/src/ and can be launched with rosrun ov_eval <binary>.
BinaryPurpose
error_singlerunATE, RPE, and NEES for a single estimated trajectory vs. ground truth
error_datasetCompare multiple algorithms on one dataset; outputs ATE, RPE, RMSE, and NEES
error_comparisonCross-dataset, cross-algorithm comparison; generates a LaTeX table
error_simulationError evaluation against a simulated ground-truth trajectory
BinaryPurpose
plot_trajectoriesPlot 2-D (XY) and altitude-vs-time trajectories for visual inspection
format_converterConvert EuRoC / ASL CSV ground-truth files to the RPE text format
live_align_trajectoryReal-time trajectory alignment for live debugging
pose_to_fileROS node that records PoseWithCovarianceStamped / Odometry topics to a text file
BinaryPurpose
timing_flamegraphFlame-graph style breakdown of per-component execution times
timing_histogramDistribution of execution times as a binned histogram
timing_comparisonSide-by-side comparison of timing files (e.g., mono vs. stereo)
timing_percentagesCPU and memory usage from psutil logs

​
Python scripts

Two Python helper scripts live in ov_eval/python/:
  • pid_ros.py — A ROS node that uses psutil to poll one or more running ROS nodes for their CPU percentage, memory percentage, and thread count, writing results to a timestamped log file.
  • pid_sys.py — A system-level variant that monitors processes by name rather than by ROS node graph.

​
Evaluation workflow

1

Run OpenVINS and record the trajectory

Add the pose_to_file recorder node to your launch file. It subscribes to any PoseWithCovarianceStamped, PoseStamped, TransformStamped, or Odometry topic and writes a timestamped text file.
<node name="recorder_estimate" pkg="ov_eval" type="pose_to_file" output="screen">
    <param name="topic"      type="str" value="/ov_msckf/poseimu" />
    <param name="topic_type" type="str" value="PoseWithCovarianceStamped" />
    <param name="output"     type="str" value="/data/traj_log.txt" />
</node>
2

Convert ground-truth to the RPE format

Ground-truth files must follow the RPE text format: time(s) px py pz qx qy qz qw. EuRoC/ASL CSV files use nanosecond timestamps and a different quaternion order; the format_converter binary handles the conversion automatically.
# Convert a single file
rosrun ov_eval format_converter groundtruth.csv

# Convert all CSV files in a folder
rosrun ov_eval format_converter /path/to/truth_folder/
3

Organise files into the expected folder structure

ov_eval expects ground-truth files and algorithm run files to be separated into two top-level directories. Dataset names must match between the truth/ and algorithms/ trees.
truth/
    V1_01_easy.txt
    V1_02_medium.txt
algorithms/
    open_vins/
        V1_01_easy/
            run1.txt
            run2.txt
        V1_02_medium/
            run1.txt
    okvis_stereo/
        V1_01_easy/
            run1.txt
4

Evaluate a single run

rosrun ov_eval error_singlerun posyaw truths/V1_01_easy.txt 1565371553_estimate.txt
This prints ATE, RPE (at multiple segment lengths), and NEES to the console, and displays inline matplotlib figures if Python libraries are available.
5

Compare algorithms across datasets

# All algorithms on one dataset
rosrun ov_eval error_dataset posyaw truths/V1_01_easy.txt algorithms/

# All algorithms across all datasets (outputs a LaTeX table)
rosrun ov_eval error_comparison posyaw truths/ algorithms/

​
Alignment modes

All error evaluation binaries accept an <align_mode> argument that controls how the estimated trajectory is registered to ground truth before error is computed:
ModeDescription
posyawAlign position and yaw only (gravity-aligned systems)
posyawsingleSame, but using only the first pose
se3Full 6-DOF rigid-body alignment
se3singleFull 6-DOF using only the first pose
sim3Similarity transform (position, rotation, and scale)
noneNo alignment; trajectories must already share a common frame
For most VIO systems running in a gravity-aligned frame, posyaw is the appropriate alignment mode. Use sim3 only for monocular systems that are scale-ambiguous.

​
Next steps

Accuracy metrics

Mathematical definitions of ATE, RPE, RMSE, and NEES, with example command usage.

Timing analysis

How to profile OpenVINS components and interpret CPU and memory usage data.

Build docs developers (and LLMs) love

Get started for freeTalk to us