English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The recent surge of interest surrounding Multimodal Neural Networks (MM-NN)is attributed to their ability to effectively process and integrate multiscaleinformation from diverse data sources. MM-NNs extract and fuse features frommultiple modalities using adequate unimodal backbones and specific fusionnetworks. Although this helps strengthen the multimodal informationrepresentation, designing such networks is labor-intensive. It requires tuningthe architectural parameters of the unimodal backbones, choosing the fusingpoint, and selecting the operations for fusion. Furthermore, multimodality AIis emerging as a cutting-edge option in Internet of Things (IoT) systems whereinference latency and energy consumption are critical metrics in addition toaccuracy. In this paper, we propose Harmonic-NAS, a framework for the jointoptimization of unimodal backbones and multimodal fusion networks with hardwareawareness on resource-constrained devices. Harmonic-NAS involves a two-tieroptimization approach for the unimodal backbone architectures and fusionstrategy and operators. By incorporating the hardware dimension into theoptimization, evaluation results on various devices and multimodal datasetshave demonstrated the superiority of Harmonic-NAS over state-of-the-artapproaches achieving up to 10.9% accuracy improvement, 1.91x latency reduction,and 2.14x energy efficiency gain.

Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devices