English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Pattern recognition through the fusion of RGB frames and Event streams hasemerged as a novel research area in recent years. Current methods typicallyemploy backbone networks to individually extract the features of RGB frames andevent streams, and subsequently fuse these features for pattern recognition.However, we posit that these methods may suffer from key issues like sematicgaps and small-scale backbone networks. In this study, we introduce a novelpattern recognition framework that consolidates the semantic labels, RGBframes, and event streams, leveraging pre-trained large-scale vision-languagemodels. Specifically, given the input RGB frames, event streams, and all thepredefined semantic labels, we employ a pre-trained large-scale vision model(CLIP vision encoder) to extract the RGB and event features. To handle thesemantic labels, we initially convert them into language descriptions throughprompt engineering, and then obtain the semantic features using the pre-trainedlarge-scale language model (CLIP text encoder). Subsequently, we integrate theRGB/Event features and semantic features using multimodal Transformer networks.The resulting frame and event tokens are further amplified using self-attentionlayers. Concurrently, we propose to enhance the interactions between texttokens and RGB/Event tokens via cross-attention. Finally, we consolidate allthree modalities using self-attention and feed-forward layers for recognition.Comprehensive experiments on the HARDVS and PokerEvent datasets fullysubstantiate the efficacy of our proposed SAFE model. The source code will bemade available at https://github.com/Event-AHU/SAFE_LargeVLM.

Semantic-Aware Frame-Event Fusion based Pattern Recognition via Large Vision-Language Models