English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

A model is considered well-calibrated when its probability estimate alignswith the actual likelihood of the output being correct. Calibrating languagemodels (LMs) is crucial, as it plays a vital role in detecting and mitigatinghallucinations, a common issue of LMs, as well as building more trustworthymodels. Yet, popular neural model calibration techniques are not well-suitedfor LMs due to their lack of flexibility in discerning answer correctness andtheir high computational costs. For instance, post-processing methods liketemperature scaling are often unable to reorder the candidate generations.Moreover, training-based methods require finetuning the entire model, which isimpractical due to the increasing sizes of modern LMs. In this paper, wepresent LitCab, a lightweight calibration mechanism consisting of a singlelinear layer taking the input text representation and manipulateing the LMoutput logits. LitCab improves model calibration by only adding < 2% of theoriginal model parameters. For evaluation, we construct CaT, a benchmarkconsisting of 7 text generation tasks, covering responses ranging from shortphrases to paragraphs. We test LitCab with Llama2-7B, where it improvescalibration across all tasks, by reducing the average ECE score by 20%. Wefurther conduct a comprehensive evaluation with 7 popular open-sourced LMs fromGPT and LLaMA families, yielding the following key findings: (1) Larger modelswithin the same family exhibit better calibration on tasks with shortgeneration tasks, but not necessarily for longer ones. (2) GPT-family modelsshow superior calibration compared to LLaMA, Llama2 and Vicuna models despitehaving much fewer parameters. (3) Finetuning pretrained model (e.g., LLaMA)with samples of limited purpose (e.g., conversations) may lead to worsecalibration, highlighting the importance of finetuning setups for calibratingLMs.

LitCab: Lightweight Calibration of Language Models on Outputs of Varied Lengths