English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vision Language Models (VLMs), which extend Large Language Models (LLM) byincorporating visual understanding capability, have demonstrated significantadvancements in addressing open-ended visual question-answering (VQA) tasks.However, these models cannot accurately interpret images infused with text, acommon occurrence in real-world scenarios. Standard procedures for extractinginformation from images often involve learning a fixed set of query embeddings.These embeddings are designed to encapsulate image contexts and are later usedas soft prompt inputs in LLMs. Yet, this process is limited to the token count,potentially curtailing the recognition of scenes with text-rich context. Toimprove upon them, the present study introduces BLIVA: an augmented version ofInstructBLIP with Visual Assistant. BLIVA incorporates the query embeddingsfrom InstructBLIP and also directly projects encoded patch embeddings into theLLM, a technique inspired by LLaVA. This approach assists the model to captureintricate details potentially missed during the query decoding process.Empirical evidence demonstrates that our model, BLIVA, significantly enhancesperformance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQAbenchmark) and in undertaking general (not particularly text-rich) VQAbenchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), and achieved17.72% overall improvement in a comprehensive multimodal LLM benchmark (MME),comparing to our baseline InstructBLIP. BLIVA demonstrates significantcapability in decoding real-world images, irrespective of text presence. Todemonstrate the broad industry applications enabled by BLIVA, we evaluate themodel using a new dataset comprising YouTube thumbnails paired withquestion-answer sets across 11 diverse categories. Our code and models arefreely accessible at https://github.com/mlpc-ucsd/BLIVA.

BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions