English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While self-supervised learning has made rapid advances in natural languageprocessing, it remains unclear when researchers should engage inresource-intensive domain-specific pretraining (domain pretraining). The law,puzzlingly, has yielded few documented instances of substantial gains to domainpretraining in spite of the fact that legal language is widely seen to beunique. We hypothesize that these existing results stem from the fact thatexisting legal NLP tasks are too easy and fail to meet conditions for whendomain pretraining can help. To address this, we first present CaseHOLD (CaseHoldings On Legal Decisions), a new dataset comprised of over 53,000+ multiplechoice questions to identify the relevant holding of a cited case. This datasetpresents a fundamental task to lawyers and is both legally meaningful anddifficult from an NLP perspective (F1 of 0.4 with a BiLSTM baseline). Second,we assess performance gains on CaseHOLD and existing legal NLP datasets. Whilea Transformer architecture (BERT) pretrained on a general corpus (Google Booksand Wikipedia) improves performance, domain pretraining (using corpus ofapproximately 3.5M decisions across all courts in the U.S. that is larger thanBERT's) with a custom legal vocabulary exhibits the most substantialperformance gains with CaseHOLD (gain of 7.2% on F1, representing a 12%improvement on BERT) and consistent performance gains across two other legaltasks. Third, we show that domain pretraining may be warranted when the taskexhibits sufficient similarity to the pretraining corpus: the level ofperformance increase in three legal tasks was directly tied to the domainspecificity of the task. Our findings inform when researchers should engageresource-intensive pretraining and show that Transformer-based architectures,too, learn embeddings suggestive of distinct legal language.

When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset