English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, there has been a surge in the popularity of pre trained largelanguage models (LLMs) (such as GPT-4), sweeping across the entire NaturalLanguage Processing (NLP) and Computer Vision (CV) communities. These LLMs havedemonstrated advanced multi-modal understanding capabilities and showcasedstrong performance across various benchmarks. The LLM has started to embodytraits of artificial general intelligence, which holds vital guidance forenhancing brain-like characteristics within visual encoding models. Hence, Thispaper proposes a new multi-modal training paradigm, aligning with LLM, forencoding fMRI activity in visual cortex. Based on this paradigm, we trained anencoding model in fMRI data named the LLM-Visual Encoding Model (LLM-VEM).Specifically, we utilize LLM (miniGPT4) to generate descriptive text for allstimulus images, forming a high-quality textual description set. Moreover, weuse the pre-trained text encoder (CLIP) to process these detailed descriptions,obtaining the text embedding features. Next, we use the contrast loss functionto minimize the distance between the image embedding features and the textembedding features to complete the alignment operation of the stimulus imageand text information. With the assistance of the pre-trained LLM, thisalignment process facilitates better learning of the visual encoding model,resulting in higher precision. The final experimental results indicate that ourtraining paradigm has significantly aided in enhancing the performance of thevisual encoding model.

Aligned with LLM: a new multi-modal training paradigm for encoding fMRI activity in visual cortex