Open AccessEnhancing semantics with multi‐objective reinforcement learning for video description
Author(s) -
Li Qinyu,
Yang Longyu,
Tang Pengjie,
Wang Hanli
Publication year - 2021
Publication title -
electronics letters
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.375
H-Index - 146
eISSN - 1350-911X
pISSN - 0013-5194
DOI - 10.1049/ell2.12334
Subject(s) - computer science , reinforcement learning , benchmark (surveying) , semantics (computer science) , consistency (knowledge bases) , artificial intelligence , sentence , task (project management) , word (group theory) , closed captioning , machine learning , natural language processing , image (mathematics) , programming language , linguistics , philosophy , management , geodesy , economics , geography
Video description is challenging due to the high complexity of translating visual content into language. In most popular attention‐based pipelines for this task, visual features and previously generated words are usually concatenated as a vector to predict the current word. However, the errors caused by the inaccuracy of the predicted words may be accumulated, and the gap between visual features and language features may bring noises into the description model. Facing these problems, a variant of recurrent neural network is designed in this work, and a novel framework is developed to enhance the visual clues for video description. Moreover, a multi‐objective reinforcement learning strategy is implemented to build a more comprehensive reward with multiple metrics to improve the consistency and semantics of the generated description sentence. The experiments on the benchmark MSR‐VTT2016 and MSVD datasets demonstrate the effectiveness of the proposed approach.