SCALE-EM
Self-supervised embeddings for endoscopic matching
- PyTorch
- Contrastive Learning
- Endoscopy
- venue
- IEEE TMI 2026
- year
- 2026
Abstract
Accurate spatial understanding is essential for image-guided surgery, augmented reality integration, and context awareness. In minimally invasive procedures, establishing precise pixel-level correspondences between endoscopic frames is critical for 3D reconstruction, camera tracking, and scene interpretation. However, weak perspective cues, non-Lambertian tissue reflections, and deformable anatomy degrade conventional computer vision techniques. This work presents a deep learning pipeline for endoscopic feature matching alongside a self-supervised optimization framework: a novel-view synthesis pipeline generates ground-truth inlier correspondences used to mine triplets within a contrastive learning paradigm, augmenting a frozen DINOv2 backbone with an additional Transformer layer optimized for cosine-similarity matching.
Introduction
Minimally invasive surgery has revolutionized modern practice by reducing patient trauma and recovery times, but restricted vision and limited tactile feedback increase technical complexity. Real-time computer vision on the endoscopic feed can provide AR overlays, tool tracking, and 3D reconstruction, all of which depend on robust feature matching between frames. The surgical environment poses unique challenges: non-rigid tissue deformation, dynamic lighting, specular reflections, smoke, and blood make general-purpose descriptors insufficient. This investigation adapts pretrained vision transformer representations to produce matchable patch embeddings for endoscopic imagery without manual annotation.
Methodology
The pipeline augments a frozen DINOv2 backbone Φ with a trainable adaptation layer Ψ. Pseudo-ground-truth correspondences are generated via novel-view synthesis: a source frame is back-projected into 3D using a monocular depth estimate, transformed with a randomly sampled rigid motion, and re-projected to synthesize a warped target view; pixel mappings between source and warped views provide geometrically consistent supervision. Triplets (anchor, positive, negative) are mined in embedding space and optimized with contrastive (triplet) loss so matching points lie close while non-matches are pushed apart. At inference, correspondences are extracted via cosine similarity with mutual nearest-neighbor matching and refined to sub-pixel precision with Fourier phase correlation.
Results
On the SCARED stereo endoscopy test split, SCALE-EM achieves the lowest mean epipolar error among all tested methods (8.63 px, ~17% better than the next-best classical descriptor) and the fewest spurious matches under a cross-sequence robustness test (160 inliers on non-corresponding images vs. thousands for LoFTR and SIFT variants). Ablation studies confirm gains from the adaptation layer Ψ and sub-pixel refinement. The method remains competitive in runtime with other learning-based matchers while prioritizing precision over match density through a conservative scoring mechanism.
Conclusion
SCALE-EM shows that self-supervised contrastive adaptation of foundation-model features can yield precise endoscopic correspondences without manual labels. The approach is most suitable for applications requiring high-precision sparse matching; conservative rejection of ambiguous matches limits density compared to some baselines. Performance depends on the quality of depth maps used for novel-view synthesis, suggesting future work on more robust geometric priors for surgical scenes.
Collaborators
- Elena De Momi