English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large pre-trained language models help to achieve state of the art on avariety of natural language processing (NLP) tasks, nevertheless, they stillsuffer from forgetting when incrementally learning a sequence of tasks. Toalleviate this problem, recent works enhance existing models by sparseexperience replay and local adaption, which yield satisfactory performance.However, in this paper we find that pre-trained language models like BERT havea potential ability to learn sequentially, even without any sparse memoryreplay. To verify the ability of BERT to maintain old knowledge, we adopt andre-finetune single-layer probe networks with the parameters of BERT fixed. Weinvestigate the models on two types of NLP tasks, text classification andextractive question answering. Our experiments reveal that BERT can actuallygenerate high quality representations for previously learned tasks in a longterm, under extremely sparse replay or even no replay. We further introduce aseries of novel methods to interpret the mechanism of forgetting and how memoryrehearsal plays a significant role in task incremental learning, which bridgesthe gap between our new discovery and previous studies about catastrophicforgetting.

Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study