English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Generative language models are usually pretrained on large text corpus viapredicting the next token (i.e., sub-word/word/phrase) given the previous ones.Recent works have demonstrated the impressive performance of large generativelanguage models on downstream tasks. However, existing generative languagemodels generally neglect an inherent challenge in text corpus during training,i.e., the imbalance between frequent tokens and infrequent ones. It can lead alanguage model to be dominated by common and easy-to-learn tokens, therebyoverlooking the infrequent and difficult-to-learn ones. To alleviate that, wepropose a MiLe Loss function for mitigating the bias of learning difficultieswith tokens. During training, it can dynamically assess the learning difficultyof a to-be-learned token, according to the information entropy of thecorresponding predicted probability distribution over the vocabulary. Then itscales the training loss adaptively, trying to lead the model to focus more onthe difficult-to-learn tokens. On the Pile dataset, we train generativelanguage models at different scales of 468M, 1.2B, and 6.7B parameters.Experiments reveal that models incorporating the proposed MiLe Loss can gainconsistent performance improvement on downstream benchmarks.

MiLe Loss: a New Loss for Mitigating the Bias of Learning Difficulties in Generative Language Models