Modulating Bottom-Up and Top-Down Visual Processing via Language-Conditional Filters

Creative Commons License

Kesen I., Can O. A., Erdem E., Erdem A., Yuret D.

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Louisiana, United States Of America, 18 - 24 June 2022, pp.4609-4619 identifier identifier

  • Publication Type: Conference Paper / Full Text
  • Doi Number: 10.1109/cvprw56347.2022.00507
  • City: Louisiana
  • Country: United States Of America
  • Page Numbers: pp.4609-4619
  • Hacettepe University Affiliated: Yes


How to best integrate linguistic and perceptual processing in multi-modal tasks that involve language and vision is an important open problem. In this work, we argue that the common practice of using language in a topdown manner, to direct visual attention over high-level visual features, may not be optimal. We hypothesize that the use of language to also condition the bottom-up processing from pixels to high-level features can provide benefits to the overall performance. To support our claim, we propose a U-Net-based model and perform experiments on two language-vision dense-prediction tasks: referring expression segmentation and language-guided image colorization. We compare results where either one or both of the top-down and bottom-up visual branches are conditioned on language. Our experiments reveal that using language to control the filters for bottom-up visual processing in addition to top-down attention leads to better results on both tasks and achieves competitive performance. Our linguistic analysis suggests that bottom-up conditioning improves segmentation of objects especially when input text refers to low-level visual concepts.