English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The task of empowering large language models (LLMs) to accurately expresstheir confidence, referred to as confidence elicitation, is essential inensuring reliable and trustworthy decision-making processes. Previous methods,which primarily rely on model logits, have become less suitable for LLMs andeven infeasible with the rise of closed-source LLMs (e.g., commercialized LLMAPIs). This leads to a growing need to explore the untapped area of\emph{non-logit-based} approaches to estimate the uncertainty of LLMs. Hence,in this study, we investigate approaches for confidence elicitation that do notrequire model fine-tuning or access to proprietary information. We introducethree categories of methods: verbalize-based, consistency-based, and theirhybrid methods for benchmarking, and evaluate their performance across fivetypes of datasets and four widely-used LLMs. Our analysis of these methodsuncovers several key insights: 1) LLMs often exhibit a high degree ofoverconfidence when verbalizing their confidence; 2) Prompting strategies suchas CoT, Top-K and Multi-step confidences improve calibration of verbalizedconfidence; 3) Consistency-based methods outperform the verbalized confidencesin most cases, with particularly notable improvements on the arithmeticreasoning task; 4) Hybrid methods consistently deliver the best performanceover their baselines, thereby emerging as a promising state-of-the-artapproach; 5) Despite these advancements, all investigated methods continue tostruggle with challenging tasks, such as those requiring professionalknowledge, leaving significant scope for improvement of confidence elicitation.

Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs