English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

If a Large Language Model (LLM) answers "yes" to the question "Are mountainstall?" then does it know what a mountain is? Can you rely on it respondingcorrectly or incorrectly to other questions about mountains? The success ofLarge Language Models (LLMs) indicates they are increasingly able to answerqueries like these accurately, but that ability does not necessarily imply ageneral understanding of concepts relevant to the anchor query. We proposeconceptual consistency to measure a LLM's understanding of relevant concepts.This novel metric measures how well a model can be characterized by finding outhow consistent its responses to queries about conceptually relevant backgroundknowledge are. To compute it we extract background knowledge by traversingpaths between concepts in a knowledge base and then try to predict the model'sresponse to the anchor query from the background knowledge. We investigate theperformance of current LLMs in a commonsense reasoning setting using the CSQAdataset and the ConceptNet knowledge base. While conceptual consistency, likeother metrics, does increase with the scale of the LLM used, we find thatpopular models do not necessarily have high conceptual consistency. Ouranalysis also shows significant variation in conceptual consistency acrossdifferent kinds of relations, concepts, and prompts. This serves as a steptoward building models that humans can apply a theory of mind to, and thusinteract with intuitively.

Unpacking Large Language Models with Conceptual Consistency