Open AccessATTENTION BASED IMAGE CAPTIONING USING DEEP LEARNING
Author(s) -
Tejaswini Nakirekanti,
D. Deepika
Publication year - 2021
Publication title -
international journal of innovative research in advanced engineering
Language(s) - English
Resource type - Journals
ISSN - 2349-2163
DOI - 10.26562/ijirae.2021.v0812.009
Subject(s) - closed captioning , computer science , artificial intelligence , encoder , image (mathematics) , natural language processing , cross entropy , deep learning , categorical variable , entropy (arrow of time) , process (computing) , natural language , pattern recognition (psychology) , machine learning , physics , quantum mechanics , operating system
The process of deriving written descriptions from a picture based on the items and behaviours portrayed is known as image caption generation. The resulting description will represent what is in the image and the interactions among the objects. However, like with any other image processing challenge, recreating this characteristic in an artificial system is a difficult feat, and so the adoption Deep Learning procedures can help to tackle the problem. The main goal of this project is to generate an image captioning model that can provide a concise and relatable description of the picture by incorporating an attention mechanism. Ideally, the model can be separated into two components – one is called Encoder and the other is a Decoder. The encoder used here is a Google InceptionV3 pre-trained model. And the decoder is a language-based model, Gated Recurrent Unit (GRU) to translate the characteristics and objects provided by the encoder into natural sentences. Furthermore, the model's effectiveness is aided by the attention mechanism. To conclude, the model is trained on a subset of the MS COCO dataset, which contains about 80,000 pictures with at least five descriptions each and evaluated based on the BLEU score. To reduce the loss, the model weights are tuned using the categorical cross-entropy loss function. The results of the model are promising and competitive.