English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Neural Machine Translation (NMT) is an open vocabulary problem. As a result,dealing with the words not occurring during training (a.k.a. out-of-vocabulary(OOV) words) have long been a fundamental challenge for NMT systems. Thepredominant method to tackle this problem is Byte Pair Encoding (BPE) whichsplits words, including OOV words, into sub-word segments. BPE has achievedimpressive results for a wide range of translation tasks in terms of automaticevaluation metrics. While it is often assumed that by using BPE, NMT systemsare capable of handling OOV words, the effectiveness of BPE in translating OOVwords has not been explicitly measured. In this paper, we study to what extentBPE is successful in translating OOV words at the word-level. We analyze thetranslation quality of OOV words based on word type, number of segments,cross-attention weights, and the frequency of segment n-grams in the trainingdata. Our experiments show that while careful BPE settings seem to be fairlyuseful in translating OOV words across datasets, a considerable percentage ofOOV words are translated incorrectly. Furthermore, we highlight the slightlyhigher effectiveness of BPE in translating OOV words for special cases, such asnamed-entities and when the languages involved are linguistically close to eachother.

How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?