English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Multimodal Large Language Models (MLLMs) have shown impressive abilities ininteracting with visual content with myriad potential downstream tasks.However, even though a list of benchmarks has been proposed, the capabilitiesand limitations of MLLMs are still not comprehensively understood, due to alack of a standardized and holistic evaluation framework. To this end, wepresent the first Comprehensive Evaluation Framework (ChEF) that canholistically profile each MLLM and fairly compare different MLLMs. First, westructure ChEF as four modular components, i.e., Scenario as scalablemultimodal datasets, Instruction as flexible instruction retrieving formulae,Inferencer as reliable question answering strategies, and Metric as indicativetask-specific score functions. Based on them, ChEF facilitates versatileevaluations in a standardized framework, and new evaluations can be built bydesigning new Recipes (systematic selection of these four components). Notably,current MLLM benchmarks can be readily summarized as recipes of ChEF. Second,we introduce 6 new recipes to quantify competent MLLMs' desired capabilities(or called desiderata, i.e., calibration, in-context learning, instructionfollowing, language performance, hallucination, and robustness) as reliableagents that can perform real-world multimodal interactions. Third, we conduct alarge-scale evaluation of 9 prominent MLLMs on 9 scenarios and 6 desiderata.Our evaluation summarized over 20 valuable observations concerning thegeneralizability of MLLMs across various scenarios and the composite capabilityof MLLMs required for multimodal interactions. We will publicly release all thedetailed implementations for further analysis, as well as an easy-to-usemodular toolkit for the integration of new recipes and models, so that ChEF canbe a growing evaluation framework for the MLLM community.

ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models