AR
Back to research
Spectra-BREAST cover

Spectra-BREAST

3D Hyperspectral Mapping of Lumpectomy Specimens

venue
Ital-IA 2026
year
2026
CodeProject

Abstract

Hyperspectral imaging (HSI) is promising for intraoperative assessment of resection margins in breast-conserving surgery, but clinical translation requires aligning 2D spectral information onto the 3D shape of excised tissue so suspicious regions can be precisely localized. We present a fully automated, calibration-free pipeline that produces a 3D hyperspectral point cloud of an ex-vivo lumpectomy specimen from consumer-camera RGB images and a single top-down HSI acquisition. Evaluated on two specimens, the pipeline achieves median 3D registration error below 1 mm and 2D reprojection error below 0.02 mm, with per-specimen processing under 4 minutes.

Introduction

Breast-conserving surgery is standard for early-stage breast cancer, but positive margins occur in 10–70% of cases and drive reoperation in ~20% of patients. Margin assessment currently relies on postoperative histopathology, while intraoperative alternatives are slower or less accurate for whole-specimen evaluation. HSI has shown high classification accuracy on breast tissue, yet existing approaches produce only 2D spectral maps, neglecting 3D geometry that clinicians use when deciding which regions require targeted follow-up.

Methodology

The pipeline inputs N RGB images and one top-down HSI cube acquired with four planar ArUco markers around the specimen. 3D geometry is recovered with a deep-learning Structure-from-Motion backbone (MASt3R-SfM) and stabilized in a metric frame via custom bundle adjustment enforcing consistency on 16 ArUco corners. HSI registration avoids recovering the HSI camera pose: shared markers define corner correspondences that drive a planar homography, and 3D coordinates are recovered by lookup on an orthographically rendered depth map. The output is a spectral point cloud C ∈ ℝ^(S×(B+3)) with metric coordinates and full spectral signatures per surface point.

Spectra-BREAST, Figure 1
Fig. 1. The registration pipeline: N RGB images and one HSI cube, four planar ArUco markers, MASt3R-SfM geometry, ray-casting onto the mesh, and a 2D-2D registration with depth lookup.

Results

On specimens SB019 and SB020, 2D reprojection error stays below 0.02 mm (mean ± std: 0.017 ± 0.009 and 0.013 ± 0.003 mm over 16 corners). Median 3D registration error is 0.945 mm [IQR 0.731–1.378] for SB019 and 0.974 mm [IQR 0.753–1.088] for SB020. End-to-end processing completes in under 4 minutes per specimen (3D reconstruction ~3 min; 2D–3D registration ~24 s on an NVIDIA A100), supporting intraoperative timing constraints.

Spectra-BREAST, Figure 2
Fig. 2. Left: 3D registration median error and total compute time. Right: 2D median reprojection error, both as a function of the fraction of RGB images used.

Conclusion

This is, to our knowledge, the first HSI–3D fusion for lumpectomy specimens achieved without dedicated geometric hardware or cross-modal pose estimation. Sub-millimeter registration and sub-4-minute runtime support feasibility for HSI-guided spatial localization in margin assessment. Limitations include evaluation on only two specimens and ground truth derived from the same photogrammetric reconstruction; independent 3D landmark validation remains future work.

Collaborators

Institutions

Alberto Rota · Research · Contact