arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Keyu Wen

8 papers · 2020–2022

2020: 22021: 12022: 5
2020peak 5 in 20222022

Frequent co-authors

Papers

showing 1–8 of 8 RSS

Learning Dual Semantic Relations with Graph Attention for Image-Text Matching

Keyu Wen, Xiaodong Gu, Qingrong Cheng

cs.CV · · cited 127× · arXiv ↗ · more like this ·

Image-Text Matching is one major task in cross-modal information processing. The main challenge is to learn the unified visual and textual representations. Previous methods that perform well on this task primarily focus on not only the alignment between region features in images and the corresponding words in sentences, but also the alignment between relations of regions and relational words. However, the lack of joint learning of regional features and global features will cause the regional features to lose contact with the global context, leading to the mismatch with those non-object words which have global meanings in some sentences. In this work, in order to alleviate this issue, it is necessary to enhance the relations between regions and the relations between regional and global concepts to obtain a more accurate visual representation so as to be better correlated to the…

COOKIE: Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation

Keyu Wen, Jin Xia, Yuanyuan Huang +3 more

Computer Vision and Pattern Recognition · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · · cited 33× · DOI ↗ · more like this ·

There has been a recent surge of interest in cross-modal pre-training. However, existed approaches pre-train a one-stream model to learn joint vision-language representation, which suffers from calculation explosion when conducting cross-modal retrieval. In this work, we propose the Contrastive Cross-Modal Knowledge Sharing Pretraining (COOKIE) method to learn universal text-image representations. There are two key designs in it, one is the weight-sharing transformer on top of the visual and textual encoders to align text and image semantically, the other is three kinds of contrastive learning designed for sharing knowledge between different modalities. Cross-modal knowledge sharing greatly promotes the learning of unimodal representation. Experiments on multi-modal matching tasks including cross-modal retrieval, text matching, and image retrieval show the effectiveness and efficiency…

Semantic Pre-Alignment and Ranking Learning With Unified Framework for Cross-Modal Retrieval

Qingrong Cheng, Zhenshan Tan, Keyu Wen +2 more

Computer Vision and Pattern Recognition · IEEE Transactions on Circuits and Systems for Video Technology · · cited 26× · DOI ↗ · more like this ·

Cross-modal retrieval aims at retrieving highly semantic relevant information among multi-modalities. Existing cross-modal retrieval methods mainly explore the semantic consistency between image and text while rarely consider the rankings of positive instances in the retrieval results. Moreover, these methods seldom take into account the cross-interaction between image and text, which leads to the deficiency of learning their semantic relations. In this paper, we propose a Unified framework with Ranking Learning (URL) for cross-modal retrieval. The unified framework consists of three sub-networks, visual network, textual network, and interaction network. Visual network and textual network project the image feature and text feature into their corresponding hidden spaces respectively. Then, the interaction network forces the target image-text representation to align in the common space.…

Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks

Qingrong Cheng, Keyu Wen, Xiaodong Gu

cs.CV · · cited 25× · arXiv ↗ · more like this ·

Text-to-image synthesis aims to generate a photo-realistic and semantic consistent image from a specific text description. The images synthesized by off-the-shelf models usually contain limited components compared with the corresponding image and text description, which decreases the image quality and the textual-visual consistency. To address this issue, we propose a novel Vision-Language Matching strategy for text-to-image synthesis, named VLMGAN*, which introduces a dual vision-language matching mechanism to strengthen the image quality and semantic consistency. The dual vision-language matching mechanism considers textual-visual matching between the generated image and the corresponding text description, and visual-visual consistent constraints between the synthesized image and the real image. Given a specific text description, VLMGAN* firstly encodes it into textual features and…

High-speed rail services for elderly passengers: Ticket-booking patterns and policy implications

Jiemin Xie, Shuguang Zhan, S.C. Wong +3 more

Transportation · Transport Policy · · cited 19× · DOI ↗ · more like this ·

Mobility is important for the well-being of the elderly, as it enables them to maintain social connections and enjoy a good quality of life. High-speed rail (HSR) is an important transportation mode in China, and considering China's aging population, HSR operators must offer convenient services to elderly people to enhance their mobility. This study analyzed HSR ticket-booking records to examine the ticket-booking patterns of elderly passengers in two China delta regions (the Pearl River Delta and the Yangtze River Delta). The results show that male and female younger elderly passengers (i.e., those aged 60–69) differed in their HSR travel preferences (e.g., seat type and booking day), whereas these inter-gender differences were less significant among elderly passengers aged 70 and over. Furthermore, compared with the younger passengers (whose ages were less than 60 years), elderly…

A Unified Two-Stage Group Semantics Propagation and Contrastive Learning Network for Co-Saliency Detection

Zhenshan Tan, Cheng Chen, Keyu Wen +2 more

cs.CV · · cited 4× · arXiv ↗ · more like this ·

Co-saliency detection (CoSOD) aims at discovering the repetitive salient objects from multiple images. Two primary challenges are group semantics extraction and noise object suppression. In this paper, we present a unified Two-stage grOup semantics PropagatIon and Contrastive learning NETwork (TopicNet) for CoSOD. TopicNet can be decomposed into two substructures, including a two-stage group semantics propagation module (TGSP) to address the first challenge and a contrastive learning module (CLM) to address the second challenge. Concretely, for TGSP, we design an image-to-group propagation module (IGP) to capture the consensus representation of intra-group similar features and a group-to-pixel propagation module (GPP) to build the relevancy of consensus representation. For CLM, with the design of positive samples, the semantic consistency is enhanced. With the design of negative…

Dual Semantic Relationship Attention Network for Image-Text Matching

Keyu Wen, Xiaodong Gu

Computer Vision and Pattern Recognition · · cited 1× · DOI ↗ · more like this ·

Image-Text Matching is one major task in cross-modal information processing. The main challenge is to learn the unified vision and language representations. Previous methods that perform well on this task primarily focus on the region features in images corresponding to the words in sentences. However, this will cause the regional features to lose contact with the global context, leading to the mismatch with those non-object words in some sentences. In this work, in order to alleviate this problem, a novel Dual Semantic Relationship Attention Network is proposed which mainly consists of two modules, separate semantic relationship module and the joint semantic relationship module. With these two modules, different hierarchies of semantic relationships are learned simultaneously, thus promoting the image-text matching process. Quantitative experiments have been performed on MS-COCO and…

Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation Learning and Retrieval

Keyu Wen, Zhenshan Tan, Qingrong Cheng +2 more

cs.CV · · cited 0× · arXiv ↗ · more like this ·

Recently, the cross-modal pre-training task has been a hotspot because of its wide application in various down-streaming researches including retrieval, captioning, question answering and so on. However, exiting methods adopt a one-stream pre-training model to explore the united vision-language representation for conducting cross-modal retrieval, which easily suffer from the calculation explosion. Moreover, although the conventional double-stream structures are quite efficient, they still lack the vital cross-modal interactions, resulting in low performances. Motivated by these challenges, we put forward a Contrastive Cross-Modal Knowledge Sharing Pre-training (COOKIE) to grasp the joint text-image representations. Structurally, COOKIE adopts the traditional double-stream structure because of the acceptable time consumption. To overcome the inherent defects of double-stream structure as…