CYWS-3D (The Change You Want to See, Now in 3D) is a PyTorch deep learning model for detecting semantic changes between two images of the same 3D scene. Unlike naive pixel-level differencing, CYWS-3D accounts for the fact that two images of the same scene may be captured from different viewpoints, at different times, or with different camera parameters — and distinguishes genuine changes in the scene from apparent differences caused purely by perspective.Documentation Index
Fetch the complete documentation index at: https://mintlify.com/ragavsachdeva/CYWS-3D/llms.txt
Use this file to discover all available pages before exploring further.
The Change Detection Problem
When comparing two images of the same scene, a straightforward pixel or feature difference will flag any region that looks different — including areas that simply appear different because the camera has moved. A chair that has been removed from a room is a real change. The same chair appearing larger in one image because the camera is closer is not. Solving this requires answering two separate questions before any comparison is made:- Where in image 2 does each visible point from image 1 project? This is the registration problem, and the answer depends on how much 3D geometry information is available.
- After accounting for viewpoint, does anything look genuinely different? This is the change detection problem, and it is what the model’s encoder-decoder and detection head are trained to answer.
CYWS-3D Approach
CYWS-3D solves change detection in feature space rather than pixel space. The pipeline proceeds in two high-level stages: Feature alignment. Deep features are extracted from both images using a frozen DINO ViT-Base backbone. A registration module then warps the features from image 2 onto the coordinate frame of image 1 (and vice versa), using whichever geometric information is available — full camera parameters, a depth map, or 2D correspondences. A visibility mask tracks which parts of image 2 are actually visible in image 1 after warping. Change detection. The aligned feature difference is passed through a U-Net encoder-decoder with SCSE attention and fused with additional DINO features. A CenterNet detection head predicts bounding boxes for changed objects. Predictions are made at 224×224 resolution and rescaled to the original image dimensions. The model returns a pair of bounding box lists — one for each input image — so that each detected change is localised in both views.Pipeline Summary
Load images and metadata
Provide image 1 and image 2. Optionally supply depth maps, camera intrinsics, rotation and position matrices, or pre-computed 2D affine transforms, depending on the registration strategy you choose.
Extract depth and correspondences
If camera intrinsics are already provided, this step is skipped. Otherwise, SuperPoint detects keypoints in both images and SuperGlue matches them. If no depth map is provided, ZoeDepth predicts monocular depth. RANSAC filters inlier correspondences.
Register features
The FeatureRegistrationModule aligns the DINO features from image 2 into the coordinate frame of image 1 using the selected strategy: full 3D point-cloud warping via PyTorch3D, 2D affine warping via grid sampling, or identity (no warping for aligned images).
Detect changes
The feature difference is encoded by a U-Net, fused with DINO features, and decoded. A CenterNet head predicts bounding boxes at 224×224 for changed regions in both images.
Supported Input Modes
The table below summarises every combination of inputs that CYWS-3D supports. The Strategy column maps directly to theregistration_strategy field in your input_metadata.yml.
| Strategy | Depth | Camera Params | Correspondences | Notes |
|---|---|---|---|---|
3d (RGB only) | Predicted by ZoeDepth | None | SuperPoint + SuperGlue | Most flexible; no extra data required |
3d (depth provided) | Ground truth depth map | None | SuperPoint + SuperGlue | More accurate than monocular prediction |
3d (full params) | Ground truth depth map | Rotation, position, intrinsics | Not needed | Most accurate; correspondences skipped entirely |
2d (RGB only) | Not used | None | SuperPoint + SuperGlue | 2D affine alignment estimated from matches |
2d (transform provided) | Not used | transfm2d_1_to_2, transfm2d_2_to_1 | Not needed | Fastest 2D path; supply your own 3×3 matrix |
identity | Not used | None | Not needed | Images are already aligned to the same viewpoint |
Registration Strategies
Deep dive into the 3D, 2D, and identity registration modes — inputs, algorithms, and YAML examples.
Model Architecture
Explore the DINO backbone, U-Net encoder-decoder, CenterNet head, and post-processing pipeline.