Open AccessDeveloping a seq2seq neural network using visual attention to transform mathematical expressions from images to LaTeX.
Author(s) -
P. A. Vyaznikov,
I. D. Kotilevets
Publication year - 2022
Publication title -
doklady belorusskogo gosudarstvennogo universiteta informatiki i radioèlektroniki
Language(s) - English
Resource type - Journals
eISSN - 2708-0382
pISSN - 1729-7648
DOI - 10.35596/1729-7648-2021-19-8-40-44
Subject(s) - computer science , closed captioning , artificial neural network , markup language , artificial intelligence , image (mathematics) , recurrent neural network , natural language , task (project management) , network architecture , architecture , encoder , pattern recognition (psychology) , natural language processing , engineering , art , visual arts , computer security , systems engineering , xml , operating system
The paper presents the methods of development and the results of research on the effectiveness of the seq2seq neural network architecture using Visual Attention mechanism to solve the im2latex problem. The essence of the task is to create a neural network capable of converting an image with mathematical expressions into a similar expression in the LaTeX markup language. This problem belongs to the Image Captioning type: the neural network scans the image and, based on the extracted features, generates a description in natural language. The proposed solution uses the seq2seq architecture, which contains the Encoder and Decoder mechanisms, as well as Bahdanau Attention. A series of experiments was conducted on training and measuring the effectiveness of several neural network models.