Transforming Disability into Ability: An Explainable Vision-to-Voice Image Captioning Framework Using Transformer Models and Edge Computing

Image captioning is an emerging field at the intersection of computer vision and natural language processing (NLP). It has shown great potential to enhance accessibility by automatically generating descriptive text for visual content. For visually impaired individuals, such technologies can transform static visual information into meaningful audio narratives and enable them for greater independence and participation in daily life. This study introduces an explainable image captioning framework for visually impaired individuals with an ability to turn visual scenes into descriptive audio output in real-time. Using the Flickr8k dataset, we trained and tested two different models of image captioning. The first model uses a Convolutional Neural Network (CNN), specifically ResNet50, as a visual encoder and a Long Short-Term Memory (LSTM) network as a language decoder. This approach incorporates Grad-CAM so that it can visualize spatial areas affecting each word prediction, and it reaches 0.5222 and 0.1069 of BLEU-1 and BLEU-4 scores. The second and more sophisticated model has the same CNN encoder, but instead of LSTM, it uses a decoder based on Transformer with multi-head self-attention and positional encoding. An attention-based model showed a significant increase in performance to BLEU-1 and BLEU-4 scores of 0.6163 and 0.2225. It has word-level self-attention visualizations that come convenient to make it more explainable, as it shows inter-token relationships in the process of caption generation. We also suggest an edge-deployable framework of IoT-enabled wearable devices that would allow local image captioning and real-time speech generation without the use of cloud services. A combination of precision, interpretability, and deployment enables the framework to be a powerful and transparent assistive framework to be used in the real world to support visually impaired individuals.