English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent research on Large Language Models (LLMs) has led to remarkableadvancements in general NLP AI assistants. Some studies have further exploredthe use of LLMs for planning and invoking models or APIs to address moregeneral multi-modal user queries. Despite this progress, complex visual-basedtasks still remain challenging due to the diverse nature of visual tasks. Thisdiversity is reflected in two aspects: 1) Reasoning paths. For many real-lifeapplications, it is hard to accurately decompose a query simply by examiningthe query itself. Planning based on the specific visual content and the resultsof each step is usually required. 2) Flexible inputs and intermediate results.Input forms could be flexible for in-the-wild cases, and involves not only asingle image or video but a mixture of videos and images, e.g., a user-viewimage with some reference videos. Besides, a complex reasoning process willalso generate diverse multimodal intermediate results, e.g., video narrations,segmented video clips, etc. To address such general cases, we propose amulti-modal AI assistant, AssistGPT, with an interleaved code and languagereasoning approach called Plan, Execute, Inspect, and Learn (PEIL) to integrateLLMs with various tools. Specifically, the Planner is capable of using naturallanguage to plan which tool in Executor should do next based on the currentreasoning progress. Inspector is an efficient memory manager to assist thePlanner to feed proper visual information into a specific tool. Finally, sincethe entire reasoning process is complex and flexible, a Learner is designed toenable the model to autonomously explore and discover the optimal solution. Weconducted experiments on A-OKVQA and NExT-QA benchmarks, achievingstate-of-the-art results. Moreover, showcases demonstrate the ability of oursystem to handle questions far more complex than those found in the benchmarks.

AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn