English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Contrastive Language--Image Pre-training (CLIP) has shown remarkable successin learning with cross-modal supervision from extensive amounts of image--textpairs collected online. Thus far, the effectiveness of CLIP has beeninvestigated primarily in general-domain multimodal problems. This workevaluates the effectiveness of CLIP for the task of Medical Visual QuestionAnswering (MedVQA). To this end, we present PubMedCLIP, a fine-tuned version ofCLIP for the medical domain based on PubMed articles. Our experiments areconducted on two MedVQA benchmark datasets and investigate two MedVQA methods,MEVF (Mixture of Enhanced Visual Features) and QCR (Question answering viaConditional Reasoning). For each of these, we assess the merits of visualrepresentation learning using PubMedCLIP, the original CLIP, andstate-of-the-art MAML (Model-Agnostic Meta-Learning) networks pre-trained onlyon visual data. We open source the code for our MedVQA pipeline andpre-training PubMedCLIP. CLIP and PubMedCLIP achieve improvements in comparisonto MAML's visual encoder. PubMedCLIP achieves the best results with gains inthe overall accuracy of up to 3%. Individual examples illustrate the strengthsof PubMedCLIP in comparison to the previously widely used MAML networks. Visualrepresentation learning with language supervision in PubMedCLIP leads tonoticeable improvements for MedVQA. Our experiments reveal distributionaldifferences in the two MedVQA benchmark datasets that have not been imparted inprevious work and cause different back-end visual encoders in PubMedCLIP toexhibit different behavior on these datasets. Moreover, we witness fundamentalperformance differences of VQA in general versus medical domains.

Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?