English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large language models (LLMs), with their remarkable conversationalcapabilities, have demonstrated impressive performance across variousapplications and have emerged as formidable AI assistants. In view of this, itraises an intuitive question: Can we harness the power of LLMs to buildmultimodal AI assistants for visual applications? Recently, several multi-modalmodels have been developed for this purpose. They typically pre-train anadaptation module to align the semantics of the vision encoder and languagemodel, followed by fine-tuning on instruction-following data. However, despitethe success of this pipeline in image and language understanding, itseffectiveness in joint video and language understanding has not been widelyexplored. In this paper, we aim to develop a novel multi-modal foundation modelcapable of comprehending video, image, and language within a general framework.To achieve this goal, we introduce Valley, a Video Assistant with LargeLanguage model Enhanced abilitY. The Valley consists of a LLM, a temporalmodeling module, a visual encoder, and a simple projection module designed tobridge visual and textual modes. To empower Valley with video comprehension andinstruction-following capabilities, we construct a video instruction datasetand adopt a two-stage tuning procedure to train it. Specifically, we employChatGPT to facilitate the construction of task-oriented conversation dataencompassing various tasks, including multi-shot captions, long videodescriptions, action recognition, causal relationship inference, etc.Subsequently, we adopt a pre-training-then-instructions-tuned pipeline to alignvisual and textual modalities and improve the instruction-following capabilityof Valley. Qualitative experiments demonstrate that Valley has the potential tofunction as a highly effective video assistant that can make complex videounderstanding scenarios easy.

Valley: Video Assistant with Large Language model Enhanced abilitY