English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this work, we study the effect of varying the architecture and trainingdata quality on the data scaling properties of Neural Machine Translation(NMT). First, we establish that the test loss of encoder-decoder transformermodels scales as a power law in the number of training samples, with adependence on the model size. Then, we systematically vary aspects of thetraining setup to understand how they impact the data scaling laws. Inparticular, we change the following (1) Architecture and task setup: We compareto a transformer-LSTM hybrid, and a decoder-only transformer with a languagemodeling loss (2) Noise level in the training distribution: We experiment withfiltering, and adding iid synthetic noise. In all the above cases, we find thatthe data scaling exponents are minimally impacted, suggesting that marginallyworse architectures or training data can be compensated for by adding moredata. Lastly, we find that using back-translated data instead of parallel data,can significantly degrade the scaling exponent.

Data Scaling Laws in NMT: The Effect of Noise and Architecture