Top-down and bottom-up attentional multiple instance learning for still image action recognition

BAŞ Ç., İKİZLER CİNBİŞ N.

SIGNAL PROCESSING-IMAGE COMMUNICATION, vol.104, 2022 (SCI-Expanded) identifier identifier

  • Publication Type: Article / Article
  • Volume: 104
  • Publication Date: 2022
  • Doi Number: 10.1016/j.image.2022.116664
  • Journal Name: SIGNAL PROCESSING-IMAGE COMMUNICATION
  • Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Academic Search Premier, Aerospace Database, Applied Science & Technology Source, Communication Abstracts, Computer & Applied Sciences, INSPEC, Metadex, Civil Engineering Abstracts
  • Keywords: Still image action recognition, Multiple instance learning, Attention, RECOGNIZING ACTIONS, MODEL
  • Hacettepe University Affiliated: Yes

Abstract

We propose an end-to-end, top-down and bottom-up attentional deep multiple instance learning approach for still image action recognition. Our approach does not rely on neither any attribute nor object labels and/or explicitly trained object detectors. Instead, our approach simultaneously proposes image regions, chooses the action related regions with a top-down attention mechanism, weights each image region with a bottom-up attention layer to mask unrelated pixels to yield a fine-grained pixel-level action related map. Then, it embeds the selected regions into a single image-level feature descriptor and assigns an action label. We evaluate our approach on four benchmark still image action datasets: Stanford40 (Yao et al., 2011), Pascal VOC 2012 (Everingham et al., 2012) MPII (Andriluka et al., 2014) and VCOCO (Gupta and Malik, 2015). Our results demonstrate that, while increasing the overall recognition performance, our framework successfully selects action related image regions and creates pixel grade action masks.