References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: BM-NAS: Bilevel Multimodal Neural Architecture Search
(Submitted on 19 Apr 2021 (v1), last revised 2 Jan 2022 (this version, v2))
Abstract: Deep neural networks (DNNs) have shown superior performances on various multimodal learning problems. However, it often requires huge efforts to adapt DNNs to individual multimodal tasks by manually engineering unimodal features and designing multimodal feature fusion strategies. This paper proposes Bilevel Multimodal Neural Architecture Search (BM-NAS) framework, which makes the architecture of multimodal fusion models fully searchable via a bilevel searching scheme. At the upper level, BM-NAS selects the inter/intra-modal feature pairs from the pretrained unimodal backbones. At the lower level, BM-NAS learns the fusion strategy for each feature pair, which is a combination of predefined primitive operations. The primitive operations are elaborately designed and they can be flexibly combined to accommodate various effective feature fusion modules such as multi-head attention (Transformer) and Attention on Attention (AoA). Experimental results on three multimodal tasks demonstrate the effectiveness and efficiency of the proposed BM-NAS framework. BM-NAS achieves competitive performances with much less search time and fewer model parameters in comparison with the existing generalized multimodal NAS methods.
Submission history
From: Yihang Yin [view email][v1] Mon, 19 Apr 2021 15:09:49 GMT (3529kb,D)
[v2] Sun, 2 Jan 2022 16:06:43 GMT (4413kb,D)
Link back to: arXiv, form interface, contact.