English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The performance of large language models (LLMs) on existing reasoningbenchmarks has significantly improved over the past years. In response, wepresent JEEBench, a considerably more challenging benchmark dataset forevaluating the problem solving abilities of LLMs. We curate 515 challengingpre-engineering mathematics, physics and chemistry problems from the highlycompetitive IIT JEE-Advanced exam. Long-horizon reasoning on top of deepin-domain knowledge is essential for solving problems in this benchmark. Ourevaluation on various open-source and proprietary models reveals that thehighest performance, even after using techniques like self-consistency,self-refinement and chain-of-thought prompting, is less than 40%. The typicalfailure modes of GPT-4, the best model, are errors in algebraic manipulation,difficulty in grounding abstract concepts into mathematical equationsaccurately and failure in retrieving relevant domain-specific concepts. We alsoobserve that by mere prompting, GPT-4 is unable to assess risk introduced bynegative marking for incorrect answers. For this, we develop a post-hocconfidence-thresholding method over self-consistency, which enables effectiveresponse selection. We hope that our challenging benchmark will guide futurere-search in problem-solving using LLMs.

Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models