A Gated Fusion Network for Dynamic Saliency Prediction

Creative Commons License

KOÇAK A., Erdem E., Erdem A.

IEEE TRANSACTIONS ON COGNITIVE AND DEVELOPMENTAL SYSTEMS, cilt.14, sa.3, ss.995-1008, 2022 (SCI-Expanded) identifier identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 14 Sayı: 3
  • Basım Tarihi: 2022
  • Doi Numarası: 10.1109/tcds.2021.3094974
  • Dergi Adı: IEEE TRANSACTIONS ON COGNITIVE AND DEVELOPMENTAL SYSTEMS
  • Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Compendex, INSPEC
  • Sayfa Sayıları: ss.995-1008
  • Anahtar Kelimeler: Videos, Adaptation models, Predictive models, Visualization, Dynamics, Feature extraction, Logic gates, Deep saliency networks, dynamic saliency estimation, gated fusion, VISUAL-ATTENTION, INTEGRATION, QUALITY
  • Hacettepe Üniversitesi Adresli: Evet

Özet

Predicting saliency in videos is a challenging problem due to complex modeling of interactions between spatial and temporal information, especially when ever-changing, dynamic nature of videos is considered. Recently, researchers have proposed large-scale data sets and models that take advantage of deep learning as a way to understand what is important for video saliency. These approaches, however, learn to combine spatial and temporal features in a static manner and do not adapt themselves much to the changes in the video content. In this article, we introduce the gated fusion network for dynamic saliency (GFSalNet), the first deep saliency model capable of making predictions in a dynamic way via the gated fusion mechanism. Moreover, our model also exploits spatial and channelwise attention within a multiscale architecture that further allows for highly accurate predictions. We evaluate the proposed approach on a number of data sets, and our experimental analysis demonstrates that it outperforms or is highly competitive with the state of the art. Importantly, we show that it has a good generalization ability, and moreover, exploits temporal information more effectively via its adaptive fusion scheme.