A Distributed Representation Based Query Expansion Approach for Image Captioning

Yagcioglu S., ERDEM M. E., Erdem A., Cakici R.

53rd Annual Meeting of the Association-for-Computational-Linguistics (ACS) / 7th International Joint Conference on Natural Language Processing of the Asian-Federation-of-Natural-Language-Processing (IJCNLP), Beijing, China, 26 - 31 July 2015, pp.106-111 identifier identifier

  • Publication Type: Conference Paper / Full Text
  • Volume:
  • Doi Number: 10.3115/v1/p15-2018
  • City: Beijing
  • Country: China
  • Page Numbers: pp.106-111
  • Hacettepe University Affiliated: Yes


In this paper, we propose a novel query expansion approach for improving transfer-based automatic image captioning. The core idea of our method is to translate the given visual query into a distributional semantics based form, which is generated by the average of the sentence vectors extracted from the captions of images visually similar to the input image. Using three image captioning benchmark datasets, we show that our approach provides more accurate results compared to the state-of-the-art data-driven methods in terms of both automatic metrics and subjective evaluation.