Documentation Index
Fetch the complete documentation index at: https://mintlify.com/ragavsachdeva/CYWS-3D/llms.txt
Use this file to discover all available pages before exploring further.
Before the model can determine what has changed between two images, it must solve a geometric alignment problem: features extracted from image 2 must be mapped into the coordinate frame of image 1 so that corresponding regions in 3D space overlap in feature space. Without this step, any difference in camera position or angle would be indistinguishable from a genuine scene change. CYWS-3D provides three registration strategies — 3d, 2d, and identity — that cover the full spectrum from richly-annotated multi-view captures to simple aligned image pairs.
Strategy Reference
When to use
Use the 3d strategy whenever the two images are captured from different 3D viewpoints — for example, photos taken of the same room or outdoor scene from different positions or angles. This strategy warps features through 3D world coordinates using PyTorch3D point-cloud rendering, which correctly handles perspective changes, occlusions, and scene depth.Three sub-modes are available depending on how much geometric metadata you can supply.| Sub-mode | depth_map | rotation, position, intrinsics |
|---|
| RGB only | Not required (predicted) | Not required |
| Depth provided | Ground truth | Not required |
| Full camera params | Ground truth | All three required |
depth_map — a per-pixel depth image aligned with the RGB image.rotation — 3×3 camera rotation matrix.position — 3-element camera position vector.intrinsics — 3×3 camera intrinsics matrix (focal lengths, principal point).
How it works
Step 1 — Depth estimation (if needed). When no depth map is provided, ZoeDepth runs on both images to predict monocular depth. When a ground-truth depth map is available, it is used directly.Step 2 — Correspondence extraction (if needed). When camera intrinsics are provided, the rotation and position matrices are used to compute the relative transformation (Rt) between the two cameras directly — no keypoint matching is required. Otherwise, SuperPoint detects up to 1 024 keypoints in each image (after resizing the longest side to 640 px), and SuperGlue matches them using a graph neural network with 20 Sinkhorn iterations. RANSAC (500 iterations, minimum 10 inliers) then filters to a reliable inlier set in 3D world coordinates (points are unprojected using depth before RANSAC). The inliers are passed to estimate_Rt_using_points, which fits a linear 3D warp via least squares.Step 3 — Differentiable feature warping. The DifferentiableFeatureWarper lifts the feature map of image 2 into a 3D point cloud using the depth map and camera parameters, then re-renders it from the viewpoint of image 1 using PyTorch3D’s PerspectiveCameras, PointsRasterizer, and AlphaCompositor. A visibility mask is computed to track which pixels in image 1 have a valid projection from image 2.Step 4 — Feature difference. The aligned feature difference is computed as:feature_diff = visibility2 × (features_image1 − features_image2_warped)
YAML example
# input_metadata.yml — 3D strategy, full camera parameters
- image1: scene_before.jpg
image2: scene_after.jpg
registration_strategy: 3d
depth_map1: depth_before.png
depth_map2: depth_after.png
rotation1: [[1,0,0],[0,1,0],[0,0,1]]
position1: [0.0, 0.0, 0.0]
rotation2: [[0.9998,-0.0175,0.0],[0.0175,0.9998,0.0],[0.0,0.0,1.0]]
position2: [0.5, 0.0, 0.0]
intrinsics: [[525.0,0,319.5],[0,525.0,239.5],[0,0,1]]
# input_metadata.yml — 3D strategy, depth only (correspondences estimated)
- image1: scene_before.jpg
image2: scene_after.jpg
registration_strategy: 3d
depth_map1: depth_before.png
depth_map2: depth_after.png
# input_metadata.yml — 3D strategy, RGB only (depth + correspondences estimated)
- image1: scene_before.jpg
image2: scene_after.jpg
registration_strategy: 3d
When to use
Use the 2d strategy when the two images can be related by a planar affine transformation — for example, images of a flat surface like a document, a wall, or a tabletop captured from slightly different angles. This strategy warps features using 2D grid sampling rather than 3D point-cloud rendering, making it faster and free of any depth dependency.The 2d strategy does not model scene depth. If the scene has significant 3D structure or the viewpoint change involves large translations in depth, use the 3d strategy instead.
| Sub-mode | transfm2d_1_to_2 / transfm2d_2_to_1 |
|---|
| RGB only | Not required (estimated from correspondences) |
| Transform provided | Both 3×3 matrices required |
transfm2d_1_to_2 — 3×3 affine matrix mapping pixel coordinates in image 1 to image 2.transfm2d_2_to_1 — 3×3 affine matrix mapping pixel coordinates in image 2 to image 1.
How it works
Step 1 — Correspondence extraction (if needed). When pre-computed transform matrices are provided, this step is skipped entirely. Otherwise, SuperPoint detects keypoints and SuperGlue matches them (same configuration as the 3d strategy). RANSAC filters inliers in 2D image coordinates (500 iterations, minimum 10 inliers). The inlier correspondences are passed to estimate_linear_warp, which solves for the best-fit 3×3 affine transformation via least squares.Step 2 — Feature warping. The registration module calls render_features_from_points, which uses PyTorch’s grid_sample to resample the feature map of image 2 into the coordinate frame of image 1 using the estimated or provided affine matrix. The same operation is applied in reverse to warp image 1 features into image 2’s frame.Step 3 — Feature difference. Because the 2d strategy does not track occlusions, visibility is treated as 1 everywhere. The feature difference is the direct element-wise subtraction of the warped and original feature maps.YAML example
# input_metadata.yml — 2D strategy, pre-computed transforms
- image1: doc_before.jpg
image2: doc_after.jpg
registration_strategy: 2d
transfm2d_1_to_2:
- [0.998, -0.063, 12.4]
- [0.063, 0.998, -3.1]
- [0.0, 0.0, 1.0]
transfm2d_2_to_1:
- [0.998, 0.063, -11.8]
- [-0.063, 0.998, 3.3]
- [0.0, 0.0, 1.0]
# input_metadata.yml — 2D strategy, correspondences estimated automatically
- image1: doc_before.jpg
image2: doc_after.jpg
registration_strategy: 2d
When to use
Use the identity strategy when the two images are already aligned to the same viewpoint — for example, two frames from a static camera, or images that have been pre-registered externally. No geometric computation is performed; features from both images are compared directly.Identity is the simplest and fastest strategy. If your images are truly co-registered, this avoids any possibility of alignment artefacts introduced by estimated transforms.
How it works
The FeatureRegistrationModule passes both feature maps through unchanged. Visibility is set to 1 for every pixel in both images. The feature difference is computed as the direct element-wise subtraction of the two feature maps:feature_diff = features_image1 − features_image2
YAML example
# input_metadata.yml — identity strategy
- image1: frame_001.jpg
image2: frame_002.jpg
registration_strategy: identity
Batch Mixing
A single batch submitted to CYWS-3D can contain items that use different registration strategies. For example, one sample in the batch may use 3d with full camera parameters while another uses identity. The FeatureRegistrationModule dispatches each sample to the appropriate sub-module based on the registration_strategy field in its metadata, then collects and concatenates the results before passing them to the encoder-decoder. This means you do not need to split your dataset by strategy or run separate inference passes.
SuperGlue Weights
The CorrespondenceExtractor supports two sets of SuperGlue pretrained weights:
indoor (default) — trained on indoor scenes; recommended for indoor environments and general use.outdoor — trained on outdoor scenes; may perform better on images captured outside.
CorrespondenceExtractor(superglue="outdoor")
3d and 2d strategies wherever correspondence estimation is triggered.
