Multimodality-based Situational Knowledge for Obstacle Detection and Alert Generation to Enhance the Navigation Assistive Systems

Recently, Deep Learning (DL) architectures have increasingly contributed to technological advancements, particularly for people with disabilities, by improving assistive systems. By adopting advanced DL models, intelligent assistive systems have recently become powerful with object detection, image captioning, and speech generation. The navigation system requires a concise and comprehensive alert generation that depends on an efficient caption generation model to describe the obstacles in the surroundings. This work introduces a Multimodality-based Situational Knowledge-aware (MuSK) system designed for obstacle alert generation in outdoor sidewalk environments by integrating visual and non-visual cues through contextual multimodal processing. To enhance obstacle detection, a layered attention mechanism fuses multi-sensory data with the visual feature maps extracted from the multimodal images, enabling obstacle-specific knowledge learning from diverse modalities. A critical innovation of this work is the personalized attention score, computed by a custom logic gate-based approach that dynamically adjusts obstacle weights based on situational knowledge based on multi-sensory data. Consequently, the logic gate mechanism refines attention scores by filtering and prioritizing relevant obstacle cues, improving obstacle detection accuracy. Furthermore, the cross-attention modeling of multimodal cues generates real-time, context-aware captions of detected obstacles, enabling precise situational awareness and safer navigation for the visually impaired. Thus, the experimental outcome reveals that the proposed MuSK system proves an accurate obstacle alert generation while tested on benchmark datasets, emphasizing the importance of personalized attention mechanisms and multi-sensory integration in improving obstacle detection in dynamic outdoor environments.