Open AccessMBA: A Multimodal Bilinear Attention Model with Residual Connection for Abstractive Multimodal Summarization
Author(s) -
Ye Xia,
Zengying Yue,
Liu Rui-heng
Publication year - 2021
Publication title -
journal of physics. conference series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.21
H-Index - 85
eISSN - 1742-6596
pISSN - 1742-6588
DOI - 10.1088/1742-6596/1856/1/012070
Subject(s) - automatic summarization , computer science , modality (human–computer interaction) , artificial intelligence , encoder , modalities , feature (linguistics) , residual , natural language processing , natural language , natural (archaeology) , task (project management) , linguistics , philosophy , algorithm , social science , management , archaeology , sociology , economics , history , operating system
The combination of vision and natural language modalities has become an important topic in both computer vision and natural language processing research communities. Multimodal summarization has received unprecedented attention with the rapid growth of multimodal information. This paper proposes MBA which consists of pre-trained feature extractors, text encoder, image encoder, multimodal bilinear attention fusion module, and summary decoder to complete abstractive multimodal summarization task. A residual network is added to the model to enhance the textual modality information and alleviate the modality-bias problem. Experiments show that the model is better than the baseline models and performs better than text summarization methods that ignore visual modality.