English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Graphical User Interface (GUI) automation holds significant promise forassisting users with complex tasks, thereby boosting human productivity.Existing works leveraging Large Language Model (LLM) or LLM-based AI agentshave shown capabilities in automating tasks on Android and Web platforms.However, these tasks are primarily aimed at simple device usage andentertainment operations. This paper presents a novel benchmark, AssistGUI, toevaluate whether models are capable of manipulating the mouse and keyboard onthe Windows platform in response to user-requested tasks. We carefullycollected a set of 100 tasks from nine widely-used software applications, suchas, After Effects and MS Word, each accompanied by the necessary project filesfor better evaluation. Moreover, we propose an advanced Actor-Critic EmbodiedAgent framework, which incorporates a sophisticated GUI parser driven by anLLM-agent and an enhanced reasoning mechanism adept at handling lengthyprocedural tasks. Our experimental results reveal that our GUI Parser andReasoning mechanism outshine existing methods in performance. Nevertheless, thepotential remains substantial, with the best model attaining only a 46% successrate on our benchmark. We conclude with a thorough analysis of the currentmethods' limitations, setting the stage for future breakthroughs in thisdomain.

ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation