English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large Language Models~(LLMs) drive current AI breakthroughs despite verylittle being known about their internal representations, e.g., how to extract afew informative features to solve various downstream tasks. To provide apractical and principled answer, we propose to characterize LLMs from ageometric perspective. We obtain in closed form (i) the intrinsic dimension inwhich the Multi-Head Attention embeddings are constrained to exist and (ii) thepartition and per-region affine mappings of the per-layer feedforward networks.Our results are informative, do not rely on approximations, and are actionable.First, we show that, motivated by our geometric interpretation, we can bypassLlama$2$'s RLHF by controlling its embedding's intrinsic dimension throughinformed prompt manipulation. Second, we derive $7$ interpretable splinefeatures that can be extracted from any (pre-trained) LLM layer, providing arich abstract representation of their inputs. Those features alone ($224$ forMistral-7B/Llama$2$-7B and $560$ for Llama$2$-70B) are sufficient to help solvetoxicity detection, infer the domain of the prompt, and even tackle the Jigsawchallenge, which aims at characterizing the type of toxicity of variousprompts. Our results demonstrate how, even in large-scale regimes, exacttheoretical results can answer practical questions in language models. Code:\url{https://github.com/RandallBalestriero/SplineLLM}.

Characterizing Large Language Model Geometry Solves Toxicity Detection and Generation