A DETR-based evidential RGB-NIR data fusion framework for fire and smoke detection

Timely and reliable detection of fire and smoke is critical, where even a single misjudgment can lead to major safety risks. While near-infrared (NIR) imagery can complement RGB data by capturing thermal signatures invisible to standard cameras, NIR data is often limited in practice, making effective fusion a challenge. In this work, we present a decision-level fusion method based on Dempster–Shafer Theory (DST), designed to integrate sparse NIR information into a pre-trained RGB detection framework. Our pipeline processes RGB images using DETR, while a Cluster Variational Autoencoder extracts soft evidence from NIR. To bridge the gap between the two modalities, we introduce a projection mechanism that maps NIR-derived beliefs to the object class space, enabling meaningful fusion even when NIR data is scarce. We also propose a KL-divergence-based loss to guide the model in learning not just accurate labels, but also spatially coherent and uncertainty-aware bounding boxes. Experiments show that our method improves mAP by around 20% over RGB-only baselines and outperforms other fusion strategies by over 30% under unbalanced settings, demonstrating its robustness and adaptability.