Leveraging RGB-D Data with Cross-Modal Context Mining for Glass Surface Detection

Jiaying Lin*, Yuen-Hei Yeung*, Shuquan Ye, Rynson W. H. Lau
City University of Hong Kong
*Joint first authors
AAAI 2025
arXiv Code (Coming soon) Dataset (Coming soon)
teaser

The advantages of detecting glass surfaces with RGB-D images. Note that red regions in the depth images represent missing depths. These examples show that the depth map can provide a strong cue for glass surface detection. State-of-the-art methods, GSDNet and EBLNet relying only on input RGB images are not able to correctly separate the glass surfaces from the background. Through learning the cross-modal contexts and the correlation between depth-missing regions and glass surface regions, our proposed model can detect the glass surfaces accurately in all three scenes.

Abstract

Glass surfaces are becoming increasingly ubiquitous as modern buildings tend to use a lot of glass panels. This, however, poses substantial challenges to the operations of autonomous systems such as robots, self-driving cars, and drones, as the glass panels can become transparent obstacles to navigation. Existing works attempt to exploit various cues, including glass boundary context or reflections, as a prior. However, they are all based on input RGB images. We observe that the transmission of 3D depth sensor light through glass surfaces often produces blank regions in the depth maps, which can offer additional insights to complement the RGB image features for glass surface detection. In this work, we propose a large-scale RGB-D glass surface detection dataset, RGB-D GSD, for rigorous experiments and future research. It contains 3,009 images offering a wide range of real-world RGB-D glass surface categories, paired with precise annotations. Moreover, we propose a novel glass surface detection framework combining RGB and depth information, with two novel modules: a cross-modal context mining (CCM) module to adaptively learn individual and mutual context features from RGB and depth information, and a depth-missing aware attention (DAA) module to explicitly exploit spatial locations where missing depths occur to help detect the presence of glass surfaces. Experimental results show that our proposed model outperforms state-of-the-art methods.

BibTeX

@article{aaai2025_rgbdglass,
  author    = {Lin, Jiaying and Yeung, Yuen-Hei and Ye, Shuquan and Lau, Rynson W.H.},
  title     = {Leveraging RGB-D Data with Cross-Modal Context Mining for Glass Surface Detection},
  journal   = {AAAI},
  year      = {2025},
}