Open AccessIncorporating Morphological Compostions with Transformer to Improve BERT
Author(s) -
Yuncheng Song,
Shuaifei Song,
Juncheng Ge,
Menghan Zhang,
Wei Yang
Publication year - 2020
Publication title -
journal of physics. conference series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.21
H-Index - 85
eISSN - 1742-6596
pISSN - 1742-6588
DOI - 10.1088/1742-6596/1486/7/072071
Subject(s) - morpheme , transformer , computer science , artificial intelligence , natural language processing , speech recognition , engineering , voltage , electrical engineering
BERT model achieves huge performance gains by modeling words and their subwords as input units. However, it still neglects the semantic information of morpheme which has been verified in many previous works. In this paper, we propose Transformer Morpheme Model (TMM), which is based on BERT and explores the effect of morpheme. Since the process of previous works about morpheme are context-independent.TMM model adopts Transformer to process morpheme information on the input layer to overcome this problem. Experiments on MRPC task are conducted to validate the feasibility of our model. TMM model has achieved about 1% gains over BERT model on MRPC task. The results demonstrate the superiority of our method and the effectiveness of morpheme information in the BERT model.