English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

With LLMs shifting their role from statistical modeling of language toserving as general-purpose AI agents, how should LLM evaluations change?Arguably, a key ability of an AI agent is to flexibly combine, as needed, thebasic skills it has learned. The capability to combine skills plays animportant role in (human) pedagogy and also in a paper on emergence phenomena(Arora & Goyal, 2023).  This work introduces Skill-Mix, a new evaluation to measure ability tocombine skills. Using a list of $N$ skills the evaluator repeatedly picksrandom subsets of $k$ skills and asks the LLM to produce text combining thatsubset of skills. Since the number of subsets grows like $N^k$, for even modest$k$ this evaluation will, with high probability, require the LLM to producetext significantly different from any text in the training set. The paperdevelops a methodology for (a) designing and administering such an evaluation,and (b) automatic grading (plus spot-checking by humans) of the results usingGPT-4 as well as the open LLaMA-2 70B model.  Administering a version of to popular chatbots gave results that, whilegenerally in line with prior expectations, contained surprises. Sizeabledifferences exist among model capabilities that are not captured by theirranking on popular LLM leaderboards ("cramming for the leaderboard").Furthermore, simple probability calculations indicate that GPT-4's reasonableperformance on $k=5$ is suggestive of going beyond "stochastic parrot" behavior(Bender et al., 2021), i.e., it combines skills in ways that it had not seenduring training.  We sketch how the methodology can lead to a Skill-Mix based eco-system ofopen evaluations for AI capabilities of future models.

Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models