English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Instruction tuning unlocks the superior capability of Large Language Models(LLM) to interact with humans. Furthermore, recent instruction-followingdatasets include images as visual inputs, collecting responses for image-basedinstructions. However, visual instruction-tuned models cannot comprehendtextual details within images well. This work enhances the current visualinstruction tuning pipeline with text-rich images (e.g., movie posters, bookcovers, etc.). Specifically, we first use publicly available OCR tools tocollect results on 422K text-rich images from the LAION dataset. Moreover, weprompt text-only GPT-4 with recognized texts and image captions to generate 16Kconversations, each containing question-answer pairs for text-rich images. Bycombining our collected data with previous multi-modal instruction-followingdata, our model, LLaVAR, substantially improves the LLaVA model's capability ontext-based VQA datasets (up to 20% accuracy improvement) while achieving anaccuracy of 91.42% on ScienceQA. The GPT-4-based instruction-followingevaluation also demonstrates the improvement of our model on both naturalimages and text-rich images. Through qualitative analysis, LLaVAR showspromising interaction (e.g., reasoning, writing, and elaboration) skills withhumans based on the latest real-world online content that combines text andimages. We make our code/data/models publicly available athttps://llavar.github.io/.

LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding