English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent advancements in Large Language Models (LLMs) have showcased strikingresults on existing logical reasoning benchmarks, with some models evensurpassing human performance. However, the true depth of their competencies androbustness, in mathematical reasoning tasks, remains an open question. Inresponse, we develop (i) an ontology of perturbations of maths questions, (ii)a semi-automatic method of perturbation, and (iii) a dataset of perturbed mathsquestions to probe the limits of LLM capabilities in mathematical reasoningtasks. These controlled perturbations span across multiple fine dimensions ofthe structural and representational aspects of maths questions. Using GPT-4, wegenerated the MORE dataset by perturbing randomly selected five seed questionsfrom GSM8K. This process was guided by our ontology and involved a thoroughautomatic and manual filtering process, yielding a set of 216 maths problems.We conducted comprehensive evaluation of both closed-source and open-sourceLLMs on MORE. The results show a significant performance drop across all themodels against the perturbed questions. This strongly suggests that currentLLMs lack robust mathematical skills and deep reasoning abilities. Thisresearch not only identifies multiple gaps in the capabilities of currentmodels, but also highlights multiple potential directions for futuredevelopment. Our dataset will be made publicly available athttps://huggingface.co/datasets/declare-lab/GSM8k_MORE.

Stuck in the Quicksand of Numeracy, Far from AGI Summit: Evaluating LLMs' Mathematical Competency through Ontology-guided Perturbations