Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/ragavsachdeva/CYWS-3D/llms.txt

Use this file to discover all available pages before exploring further.

CorrespondenceExtractor is a self-contained nn.Module that establishes pixel-level correspondences between two images of the same 3D scene. It combines SuperPoint (learned keypoint detection and description) with SuperGlue (graph-neural-network-based matching) from the SuperGluePretrainedNetwork submodule. After matching, unreliable correspondences are rejected with a RANSAC-based filter, and the surviving keypoint pairs are written back into the batch dict as normalised coordinates. The model uses these correspondences to warp feature maps between views for change detection.
CorrespondenceExtractor depends on the SuperGluePretrainedNetwork Git submodule. Clone the repository with --recursive to include it:
git clone --recursive https://github.com/ragavsachdeva/CYWS-3D.git
If you already cloned without --recursive, run:
git submodule update --init --recursive

​
Constructor

from modules.correspondence_extractor import CorrespondenceExtractor

extractor = CorrespondenceExtractor(
    nms_radius=4,
    keypoint_threshold=0.005,
    max_keypoints=1024,
    superglue="indoor",
    sinkhorn_iterations=20,
    match_threshold=0.2,
    resize=640,
)
​
nms_radius
int
default:"4"
Non-maximum suppression radius (in pixels) applied to the SuperPoint heatmap before keypoint selection. Larger values reduce the density of keypoints and help avoid spatially clustered detections.
​
keypoint_threshold
float
default:"0.005"
Minimum SuperPoint detector score for a pixel to be retained as a keypoint. Raise this value to keep only the most confidently detected corners and blobs.
​
max_keypoints
int
default:"1024"
Hard cap on the number of keypoints extracted per image. SuperPoint returns the top-scoring keypoints up to this limit. Higher values improve matching recall but increase memory usage and runtime.
​
superglue
str
default:"\"indoor\""
SuperGlue weight variant to load. Use "indoor" for scenes captured inside buildings (trained on ScanNet) and "outdoor" for exterior scenes (trained on MegaDepth). The choice affects the learned context aggregation priors.
​
sinkhorn_iterations
int
default:"20"
Number of Sinkhorn normalisation iterations used by the SuperGlue optimal transport solver. More iterations improve the quality of the soft assignment matrix at the cost of additional computation per forward pass.
​
match_threshold
float
default:"0.2"
Minimum mutual matching score (post-Sinkhorn) for a keypoint pair to be accepted as a correspondence. Lowering this threshold increases recall but may admit more false matches before RANSAC filtering.
​
resize
int
default:"640"
Maximum side length (in pixels) to which input images are resized before being passed to SuperPoint and SuperGlue. Resizing is performed with kornia.augmentation.Resize along the longer axis, preserving aspect ratio. Larger values retain more detail but require more GPU memory.

​
forward(batch)

batch = extractor(batch)
# batch["points1"] and batch["points2"] are now populated
Processes every item in the batch that requires keypoint-based correspondence and augments the batch dict in place.

​
What the forward pass reads from batch

KeyUsed for
image1, image2Source RGB images; converted to grayscale internally before passing to SuperPoint
registration_strategyPer-item routing — see skip conditions below
intrinsics1Skip condition: if already set, correspondence extraction is bypassed for that item
transfm2d_1_to_2Skip condition: if already set, 2-D transform is available and extraction is skipped

​
Skip conditions

The extractor skips an individual batch item if any of the following are true:
  • registration_strategy[i] == "identity" — the images are assumed to be pre-aligned.
  • intrinsics1[i] is already populated — 3-D camera geometry is available, no 2-D homography is needed from correspondences.
  • transfm2d_1_to_2[i] is already set — a 2-D transform was provided in the metadata.

​
What the forward pass writes to batch

KeyShapeDescription
batch["points1"]per-item Tensor [M, 2]RANSAC-inlier keypoint coordinates in image 1, normalised to [0, 1]
batch["points2"]per-item Tensor [M, 2]Corresponding keypoint coordinates in image 2, normalised to [0, 1]
Coordinates are normalised relative to the original image dimensions (before the 640-px resize), so they remain valid after any subsequent resizing in prepare_batch_for_model.

​
RANSAC Filtering

After SuperGlue produces an initial set of matched keypoint pairs, filter_out_bad_correspondences_using_ransac is called to remove geometric outliers. The filter behaves differently depending on whether depth and camera data are available.

​
3-D mode (depth + intrinsics available)

When depth1, depth2, intrinsics1, and intrinsics2 are present the filter back-projects each matched keypoint into 3-D world space using the depth map and camera intrinsics. Correspondence quality is measured as the reprojection error after applying the relative camera transform. The median reprojection error across all matches is used as the inlier threshold.

​
2-D mode (no depth data)

When depth or intrinsics are absent, the filter estimates a planar homography between the two sets of matched points using RANSAC and classifies each match as an inlier or outlier based on the homography reprojection error.

​
Shared settings

SettingValue
RANSAC iterations500
Inlier thresholdMedian reprojection error of all candidate matches
Minimum inliers required10
If fewer than 10 RANSAC inliers are found, the extractor falls back to returning all SuperGlue matches without geometric filtering to avoid discarding all correspondences in textureless scenes.

​
Code Example

from modules.correspondence_extractor import CorrespondenceExtractor

extractor = CorrespondenceExtractor()  # uses all defaults

# Assuming `batch` was built by create_batch_from_metadata
batch = extractor(batch)

# Inspect correspondences for the first item
pts1 = batch["points1"][0]  # Tensor [M, 2], values in [0, 1]
pts2 = batch["points2"][0]  # Tensor [M, 2], values in [0, 1]
print(f"Found {pts1.shape[0]} inlier correspondences")

Build docs developers (and LLMs) love

Get started for freeTalk to us