Open AccessMultimodal Offensive Meme Classification u sing Transformers and BiLSTM
Author(s) -
Roshan Nayak,
AUTHOR_ID,
B S Ullas Kannantha,
S Kruthi,
C. Gururaj,
AUTHOR_ID,
AUTHOR_ID,
AUTHOR_ID
Publication year - 2022
Publication title -
international journal of engineering and advanced technology
Language(s) - English
Resource type - Journals
ISSN - 2249-8958
DOI - 10.35940/ijeat.c3392.0211322
Subject(s) - offensive , pooling , computer science , voting , transformer , artificial intelligence , encoder , machine learning , pattern recognition (psychology) , political science , mathematics , engineering , operations research , voltage , electrical engineering , politics , law , operating system
Nowadays memes have become a way in which people express their ideas on social media. These memes can convey various views including offensive ones. Memes can be intended for a personal attack, homophobic abuse, racial abuse, attack on minority etc. The memes are implicit and multi-modal in nature. Here we analyze the meme by categorizing them as offensive or not offensive and this becomes a binary classification problem. We propose a novel offensive meme classification using the transformer-based image encoder, BiLSTM for text with mean pooling as text encoder and a Feed-Forward Network as a classification head. The SwinT + BiLSTM has performed better when compared to the ViT + BiLSTM across all the dimensions. The performance of the models has improved significantly when the contextual embeddings from DistilBert replace the custom embeddings. We have achieved the highest recall of 0.631 by combining outputs of four models using the soft voting technique.