Open AccessMorphological Analyzer using the BILSTM Model only for Japanese Hiragana Sentences
Author(s) -
Jun Izutsu,
Kanako Komiya
Publication year - 2022
Publication title -
international journal on natural language computing (print)/international journal on natural language computing
Language(s) - English
Resource type - Journals
eISSN - 2319-4111
pISSN - 2278-1307
DOI - 10.5121/ijnlc.2022.11103
Subject(s) - computer science , natural language processing , japanese language , artificial intelligence , word (group theory) , speech recognition , linguistics , philosophy
This study proposes a method to develop neural models of the morphological analyzer for Japanese Hiragana sentences using the Bi-LSTM CRF model. Morphological analysis is a technique that divides text data into words and assigns information such as parts of speech. In Japanese natural language processing systems, this technique plays an essential role in downstream applications because the Japanese language does not have word delimiters between words. Hiragana is a type of Japanese phonogramic characters, which is used for texts for children or people who cannot read Chinese characters. Morphological analysis of Hiragana sentences is more difficult than that of ordinary Japanese sentences because there is less information for dividing. For morphological analysis of Hiragana sentences, we demonstrated the effectiveness of fine-tuning using a model based on ordinary Japanese text and examined the influence of training data on texts of various genres.