CYWS-3D is built from four sequential stages: a frozen DINO ViT-Base feature backbone, a differentiable feature registration module, a U-Net encoder-decoder with attention, and a CenterNet detection head. Each stage is designed to work at a consistent spatial resolution of 224×224, giving the model a good balance between receptive field and spatial precision for localising changed objects as bounding boxes. The full configuration driving these stages is shown below.Documentation Index
Fetch the complete documentation index at: https://mintlify.com/ragavsachdeva/CYWS-3D/llms.txt
Use this file to discover all available pages before exploring further.
Feature Backbone
Both input images are independently passed through a shared FeatureBackbone built on a pretrained DINO ViT-Base with patch size 8 (vit_base_patch8_224_dino). The backbone weights are frozen — no gradient flows back into the ViT during training.
Two modifications are made to the standard ViT-Base:
- Stride patching. The default patch stride is reduced from 8 to 4. This doubles the spatial resolution of the output feature grid (from 28×28 to 56×56 at 224×224 input), producing finer-grained spatial features without changing the patch embedding weights.
- Layer hooks. Forward hooks are registered on Transformer layers 2 and 11 (
vit_feature_layers: [2, 11]). At each hooked layer, the key vectors are extracted from the QKV attention projection. This gives two sets of features: early-layer features (layer 2) that are more spatial and texture-sensitive, and late-layer features (layer 11) that carry higher-level semantic content.
(B, N, C) — where N is the number of patch tokens — into a spatial feature map of shape (B, C, H, W). The CLS token is discarded during this reshape (keep_cls_in_seq2spatial: False). The final output_dim of the backbone is 768, matching the ViT-Base hidden size.
Input images are resized to 224×224 before being passed to the backbone. ImageNet normalisation is applied (
imagenet_normalisation: True). Bounding box predictions are made at 224×224 and rescaled back to the original image resolution by visualise_predictions().Feature Registration
The FeatureRegistrationModule aligns the features extracted from image 2 into the coordinate frame of image 1 (and vice versa) using the registration strategy specified per sample. The core component for the3d strategy is the DifferentiableFeatureWarper, which works as follows:
- Point cloud construction. The feature map is unprojected from 2D into 3D world coordinates using the depth map and camera intrinsics, forming a coloured point cloud where each point carries a feature vector instead of an RGB colour.
- Rendering. PyTorch3D’s
PerspectiveCameras,PointsRasterizer, andAlphaCompositorre-render the point cloud from the target camera’s viewpoint, producing a warped feature map aligned to the target image’s frame. - Visibility masking. During rendering, a per-pixel visibility mask is computed. Pixels in the target view that have no corresponding point from the source view are marked as invisible.
- Feature difference. The change signal is computed as:
2d strategy, warping is performed via PyTorch’s grid_sample (bilinear interpolation) using the estimated or provided 3×3 affine matrix. For the identity strategy, no warping occurs and visibility is 1 everywhere.
Encoder-Decoder
The aligned feature difference map feeds into a U-Net encoder-decoder.Encoder
FourDownSamplingBlock layers progressively reduce spatial resolution while increasing feature depth:
| Block | Input Channels | Output Channels |
|---|---|---|
| 1 | 768 | 512 |
| 2 | 512 | 512 |
| 3 | 512 | 512 |
| 4 | 512 | 512 |
Decoder
The UnetDecoder uses SCSE (Squeeze-and-Channel Spatial Excitation) attention at each upsampling stage to recalibrate feature responses both channel-wise and spatially. Decoder channel widths progressively narrow as resolution is restored:| Decoder Stage | Channels |
|---|---|
| 1 | 256 |
| 2 | 256 |
| 3 | 128 |
| 4 | 128 |
| 5 | 64 |
encoder_channels configuration ([0, 768, 512, 512, 512, 512]) describes the skip-connection sizes available to each decoder stage. The leading 0 indicates that the bottleneck has no additional skip input from outside the encoder.
Feature Fusion
Before the detection head, a FeatureFusionBlock combines the decoder output with the late-layer DINO features (layer 11) at 224×224 resolution. This re-introduces high-level semantic context from the frozen backbone into the change signal, which helps the model distinguish changed objects from background noise.Detection Head
The model uses a CenterNetHead to predict bounding boxes from the fused feature map. Key parameters:| Parameter | Value |
|---|---|
| Number of classes | 1 (changed object) |
| Top-k candidates | 100 |
| Local maximum kernel | 3×3 |
| Max detections per image | 100 |
topk=100 means the 100 highest-scoring peak candidates are considered before the per-image cap is applied.
All bounding boxes are output at 224×224 resolution in [x1, y1, x2, y2, score] format. Model.predict() returns a pair (batch_image1_predicted_bboxes, batch_image2_predicted_bboxes).
Post-Processing
After the detection head, three post-processing steps refine the raw predictions: 1. Area filtering.remove_bboxes_with_area_less_than(bboxes, threshold) uses Shapely to compute each bounding box’s area and discards any box below a minimum size. The default threshold is 400 pixels² at 224×224 resolution, which removes single-pixel noise and tiny spurious detections.
2. Greedy NMS. suppress_overlapping_bboxes(bboxes, scores, iou_threshold=0.2) applies greedy non-maximum suppression. Boxes are sorted by score; any box with IoU greater than 0.2 with a higher-scoring box is removed. The low IoU threshold keeps the suppression conservative, retaining nearby but distinct changed objects.
3. Cross-image matching. keep_matching_bboxes(batch, image_index, ...) projects the centre of each bounding box detected in image 1 into the coordinate frame of image 2 using transform_points_1_to_2, then checks whether that projected centre falls inside any bounding box in image 2. Only boxes that have a corresponding detection in the other view are retained. This enforces the constraint that a real change must be visible — and detected — in both images.