English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Breakthroughs in transformer-based models have revolutionized not only theNLP field, but also vision and multimodal systems. However, althoughvisualization and interpretability tools have become available for NLP models,internal mechanisms of vision and multimodal transformers remain largelyopaque. With the success of these transformers, it is increasingly critical tounderstand their inner workings, as unraveling these black-boxes will lead tomore capable and trustworthy models. To contribute to this quest, we proposeVL-InterpreT, which provides novel interactive visualizations for interpretingthe attentions and hidden representations in multimodal transformers.VL-InterpreT is a task agnostic and integrated tool that (1) tracks a varietyof statistics in attention heads throughout all layers for both vision andlanguage components, (2) visualizes cross-modal and intra-modal attentionsthrough easily readable heatmaps, and (3) plots the hidden representations ofvision and language tokens as they pass through the transformer layers. In thispaper, we demonstrate the functionalities of VL-InterpreT through the analysisof KD-VLP, an end-to-end pretraining vision-language multimodaltransformer-based model, in the tasks of Visual Commonsense Reasoning (VCR) andWebQA, two visual question answering benchmarks. Furthermore, we also present afew interesting findings about multimodal transformer behaviors that werelearned through our tool.

VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers