English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pretrained language models have achieved a new state of the art on many NLPtasks, but there are still many open questions about how and why they work sowell. We investigate the contextualization of words in BERT. We quantify theamount of contextualization, i.e., how well words are interpreted in context,by studying the extent to which semantic classes of a word can be inferred fromits contextualized embeddings. Quantifying contextualization helps inunderstanding and utilizing pretrained language models. We show that top layerrepresentations achieve high accuracy inferring semantic classes; that thestrongest contextualization effects occur in the lower layers; that localcontext is mostly sufficient for semantic class inference; and that top layerrepresentations are more task-specific after finetuning while lower layerrepresentations are more transferable. Finetuning uncovers task relatedfeatures, but pretrained knowledge is still largely preserved.

Quantifying the Contextualization of Word Representations with Semantic Class Probing