English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Reconstructing visual stimuli from measured functional magnetic resonanceimaging (fMRI) has been a meaningful and challenging task. Previous studieshave successfully achieved reconstructions with structures similar to theoriginal images, such as the outlines and size of some natural images. However,these reconstructions lack explicit semantic information and are difficult todiscern. In recent years, many studies have utilized multi-modal pre-trainedmodels with stronger generative capabilities to reconstruct images that aresemantically similar to the original ones. However, these images haveuncontrollable structural information such as position and orientation. Toaddress both of the aforementioned issues simultaneously, we propose atwo-stage image reconstruction model called MindDiffuser, utilizing StableDiffusion. In Stage 1, the VQ-VAE latent representations and the CLIP textembeddings decoded from fMRI are put into the image-to-image process of StableDiffusion, which yields a preliminary image that contains semantic andstructural information. In Stage 2, we utilize the low-level CLIP visualfeatures decoded from fMRI as supervisory information, and continually adjustthe two features in Stage 1 through backpropagation to align the structuralinformation. The results of both qualitative and quantitative analysesdemonstrate that our proposed model has surpassed the current state-of-the-artmodels in terms of reconstruction results on Natural Scenes Dataset (NSD).Furthermore, the results of ablation experiments indicate that each componentof our model is effective for image reconstruction.

MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion