English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformer language models (TLMs) are critical for most NLP tasks, but theyare difficult to create for low-resource languages because of how muchpretraining data they require. In this work, we investigate two techniques fortraining monolingual TLMs in a low-resource setting: greatly reducing TLM size,and complementing the masked language modeling objective with twolinguistically rich supervised tasks (part-of-speech tagging and dependencyparsing). Results from 7 diverse languages indicate that our model, MicroBERT,is able to produce marked improvements in downstream task evaluations relativeto a typical monolingual TLM pretraining approach. Specifically, we find thatmonolingual MicroBERT models achieve gains of up to 18% for parser LAS and 11%for NER F1 compared to a multilingual baseline, mBERT, while having less than1% of its parameter count. We conclude reducing TLM parameter count and usinglabeled data for pretraining low-resource TLMs can yield large quality benefitsand in some cases produce models that outperform multilingual approaches.

MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning