English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent evaluations of Large Language Models (LLMs) have centered aroundtesting their zero-shot/few-shot capabilities for basic natural language tasksand their ability to translate instructions into tool APIs. However, theevaluation of LLMs utilizing complex tools to finish multi-turn, multi-modalinstructions in a complex multi-modal environment has not been investigated. Toaddress this gap, we introduce the PowerPoint Task Completion (PPTC) benchmarkto assess LLMs' ability to create and edit PPT files based on userinstructions. It contains 279 multi-turn sessions covering diverse topics andhundreds of instructions involving multi-modal operations. We also propose thePPTX-Match Evaluation System that evaluates if LLMs finish the instructionbased on the prediction file rather than the label API sequence, thus itsupports various LLM-generated API sequences. We measure 3 closed LLMs and 6open-source LLMs. The results show that GPT-4 outperforms other LLMs with75.1\% accuracy in single-turn dialogue testing but faces challenges incompleting entire sessions, achieving just 6\% session accuracy. We find threemain error causes in our benchmark: error accumulation in the multi-turnsession, long PPT template processing, and multi-modality perception. Thesepose great challenges for future LLM and agent systems. We release the data,code, and evaluation system of PPTC at \url{https://github.com/gydpku/PPTC}.

PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion