Computer Science · Computer Vision and Pattern Recognition
Cross-Modal Video Retrieval Based on Semantic Enhancement
See on the mapAbstract
Video retrieval is a challenging cross-modal task that aims to semantically align video with text. Videos contain rich visual and audio information, but existing video retrieval methods tend to ignore the extraction of audio features, which can easily lead to miss emotional information and environmental background information in video. For text, it usually only covers a small part of the video, and there are text semantic limitations. For video, due to the diversity of video information caused by the style and theme of video content, it is difficult to focus on more important and useful information in the video. Therefore, it is difficult to accurately align text and video semantics in the common space. To address the above problems, this paper uses the complementarity and relevance of cross-modality to enrich and enhance the semantic representation capabilities of each modality (visual, audio, and text), that is, to achieve "trimodal semantic enhancement". Therefore, this paper proposes a cross-modal video retrieval model based on semantic enhancement. In particular, due to the enhancement of visual, audio, and text semantics, the association between video and text is more comprehensive and accurate. By experimentally comparing with multiple existing methods on two datasets (including MSR-VTT, MSVD), the model achieved R@1 indicators of 47.5% on MSR-VTT datasets, indicating that the proposed model can improve the efficiency of cross-modal video retrieval.