Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Modulating Bottom-Up and Top-Down Visual Processing via Language-Conditional Filters
(Submitted on 28 Mar 2020 (v1), last revised 23 Jun 2022 (this version, v3))
Abstract: How to best integrate linguistic and perceptual processing in multi-modal tasks that involve language and vision is an important open problem. In this work, we argue that the common practice of using language in a top-down manner, to direct visual attention over high-level visual features, may not be optimal. We hypothesize that the use of language to also condition the bottom-up processing from pixels to high-level features can provide benefits to the overall performance. To support our claim, we propose a U-Net-based model and perform experiments on two language-vision dense-prediction tasks: referring expression segmentation and language-guided image colorization. We compare results where either one or both of the top-down and bottom-up visual branches are conditioned on language. Our experiments reveal that using language to control the filters for bottom-up visual processing in addition to top-down attention leads to better results on both tasks and achieves competitive performance. Our linguistic analysis suggests that bottom-up conditioning improves segmentation of objects especially when input text refers to low-level visual concepts. Code is available at this https URL
Submission history
From: İlker Kesen [view email][v1] Sat, 28 Mar 2020 07:54:03 GMT (749kb,D)
[v2] Mon, 18 Oct 2021 11:30:12 GMT (1654kb,D)
[v3] Thu, 23 Jun 2022 14:02:40 GMT (1524kb,D)
Link back to: arXiv, form interface, contact.