Author

Weining Wang

94 papers · 1993–2026

1993peak 18 in 20262026

Frequent co-authors

Papers

showing 1–25 of 94 RSS

VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

Jing Liu, Sihan Chen, Xingjian He +4 more

cs.LG · April 2023 · cited 204× · arXiv ↗ · more like this ·

In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner. It contains three separate encoders for single modality representations, and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain VALOR model, including Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language and audio to the same common space, building vision-language, audio-language and audiovisual-language alignment simultaneously. MGC learns how to generate text tokens in conditions of vision, audio or their both. To promote vision-audio-language pretraining research, we construct a large-scale…

Temporal Memory Attention for Video Semantic Segmentation

Hao Wang, Weining Wang, Jing Liu

cs.CV · February 2021 · cited 97× · arXiv ↗ · more like this ·

Video semantic segmentation requires to utilize the complex temporal relations between frames of the video sequence. Previous works usually exploit accurate optical flow to leverage the temporal relations, which suffer much from heavy computational cost. In this paper, we propose a Temporal Memory Attention Network (TMANet) to adaptively integrate the long-range temporal relations over the video sequence based on the self-attention mechanism without exhaustive optical flow prediction. Specially, we construct a memory using several past frames to store the temporal information of the current frame. We then propose a temporal memory attention module to capture the relation between the current frame and the memory to enhance the representation of the current frame. Our method achieves new state-of-the-art performances on two challenging video semantic segmentation datasets, particularly…

LASSO-Driven Inference in Time and Space

Victor Chernozhukov, Wolfgang K. Härdle, Chen Huang +1 more

econ.EM · June 2018 · cited 58× · arXiv ↗ · more like this ·

We consider the estimation and inference in a system of high-dimensional regression equations allowing for temporal and cross-sectional dependency in covariates and error processes, covering rather general forms of weak temporal dependence. A sequence of regressions with many regressors using LASSO (Least Absolute Shrinkage and Selection Operator) is applied for variable selection purpose, and an overall penalty level is carefully chosen by a block multiplier bootstrap procedure to account for multiplicity of the equations and dependencies in the data. Correspondingly, oracle properties with a jointly selected tuning parameter are derived. We further provide high-quality de-biased simultaneous inference on the many target parameters of the system. We provide bootstrap consistency results of the test procedure, which are based on a general Bahadur representation for the $Z$-estimators…

Learning Disentangled Representation for One-shot Progressive Face Swapping

Qi Li, Weining Wang, Chengzhong Xu +2 more

cs.CV · March 2022 · cited 47× · arXiv ↗ · more like this ·

Although face swapping has attracted much attention in recent years, it remains a challenging problem. Existing methods leverage a large number of data samples to explore the intrinsic properties of face swapping without considering the semantic information of face images. Moreover, the representation of the identity information tends to be fixed, leading to suboptimal face swapping. In this paper, we present a simple yet efficient method named FaceSwapper, for one-shot face swapping based on Generative Adversarial Networks. Our method consists of a disentangled representation module and a semantic-guided fusion module. The disentangled representation module comprises an attribute encoder and an identity encoder, which aims to achieve the disentanglement of the identity and attribute information. The identity encoder is more flexible, and the attribute encoder contains more attribute…

Multi-caption Text-to-Face Synthesis: Dataset and Algorithm

Jianxin Sun, Qi Li, Weining Wang +2 more

Computer Vision and Pattern Recognition · 17 Oct 2021 · cited 45× · DOI ↗ · more like this ·

Text-to-Face synthesis with multiple captions is still an important yet less addressed problem because of the lack of effective algorithms and large-scale datasets. We accordingly propose a Semantic Embedding and Attention (SEA-T2F) network that allows multiple captions as input to generate highly semantically related face images. With a novel Sentence Features Injection Module, SEA-T2F can integrate any number of captions into the network. In addition, an attention mechanism named Attention for Multiple Captions is proposed to fuse multiple word features and synthesize fine-grained details. Considering text-to-face generation is an ill-posed problem, we also introduce an attribute loss to guide the network to generate sentence-related attributes. Existing datasets for text-to-face are either too small or roughly generated according to attribute labels, which is not enough to train deep…

OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation

Jing Liu, Xinxin Zhu, Fei Liu +8 more

cs.CV · 6 Jul 2021 · cited 41× · arXiv ↗ · more like this ·

In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources. OPT is constructed in an encoder-decoder framework, including three single-modal encoders to generate token-based embeddings for each modality, a cross-modal encoder to encode the correlations among the three modalities, and two cross-modal decoders to generate text and image respectively. For the OPT's pre-training, we design a multi-task pretext learning scheme to model multi-modal resources from three different data granularities, \ie, token-, modality-, and sample-level modeling, through which OPT learns to align and translate among different modalities. The pre-training task is carried out on a large amount of image-text-audio triplets from Open Images. Experimental results show that OPT can learn strong image-text-audio…

Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation

Jiawei Liu, Weining Wang, Sihan Chen +2 more

cs.CV · 29 Mar 2023 · cited 27× · arXiv ↗ · more like this ·

As a combination of visual and audio signals, video is inherently multi-modal. However, existing video generation methods are primarily intended for the synthesis of visual frames, whereas audio signals in realistic videos are disregarded. In this work, we concentrate on a rarely investigated problem of text guided sounding video generation and propose the Sounding Video Generator (SVG), a unified framework for generating realistic videos along with audio signals. Specifically, we present the SVG-VQGAN to transform visual frames and audio melspectrograms into discrete tokens. SVG-VQGAN applies a novel hybrid contrastive learning method to model inter-modal and intra-modal consistency and improve the quantized representations. A cross-modal attention module is employed to extract associated features of visual frames and audio signals for contrastive learning. Then, a Transformer-based…

MOSO: Decomposing MOtion, Scene and Object for Video Prediction

Mingzhen Sun, Weining Wang, Xinxin Zhu +1 more

cs.CV · 16 Mar 2023 · cited 26× · arXiv ↗ · more like this ·

Motion, scene and object are three primary visual components of a video. In particular, objects represent the foreground, scenes represent the background, and motion traces their dynamics. Based on this insight, we propose a two-stage MOtion, Scene and Object decomposition framework (MOSO) for video prediction, consisting of MOSO-VQVAE and MOSO-Transformer. In the first stage, MOSO-VQVAE decomposes a previous video clip into the motion, scene and object components, and represents them as distinct groups of discrete tokens. Then, in the second stage, MOSO-Transformer predicts the object and scene tokens of the subsequent video clip based on the previous tokens and adds dynamic motion at the token level to the generated object and scene tokens. Our framework can be easily extended to unconditional video generation and video frame interpolation tasks. Experimental results demonstrate that…

Learnable Feature Augmentation Framework for Temporal Action Localization

Yepeng Tang, Weining Wang, Chunjie Zhang +2 more

Computer Vision and Pattern Recognition · IEEE Transactions on Image Processing · 2024 · cited 20× · DOI ↗ · more like this ·

Temporal action localization (TAL) has drawn much attention in recent years, however, the performance of previous methods is still far from satisfactory due to the lack of annotated untrimmed video data. To deal with this issue, we propose to improve the utilization of current data through feature augmentation. Given an input video, we first extract video features with pre-trained video encoders, and then randomly mask various semantic contents of video features to consider different views of video features. To avoid damaging important action-related semantic information, we further develop a learnable feature augmentation framework to generate better views of videos. In particular, a Mask-based Feature Augmentation Module (MFAM) is proposed. The MFAM has three advantages: 1) it captures the temporal and semantic relationships of original video features, 2) it generates masked features…

An Efficient Sampling-Based Attention Network for Semantic Segmentation

Xingjian He, Jing Liu, Weining Wang +1 more

Computer Vision and Pattern Recognition · IEEE Transactions on Image Processing · 2022 · cited 19× · DOI ↗ · more like this ·

Self-attention is widely explored to model long-range dependencies in semantic segmentation. However, this operation computes pair-wise relationships between the query point and all other points, leading to prohibitive complexity. In this paper, we propose an efficient Sampling-based Attention Network which combines a novel sample method with an attention mechanism for semantic segmentation. Specifically, we design a Stochastic Sampling-based Attention Module (SSAM) to capture the relationships between the query point and a stochastic sampled representative subset from a global perspective, where the sampled subset is selected by a Stochastic Sampling Module. Compared to self-attention, our SSAM achieves comparable segmentation performance while significantly reducing computational redundancy. In addition, with the observation that not all pixels are interested in the contextual…

Face Sketch Synthesis via Semantic-Driven Generative Adversarial Network

Xingqun Qi, Muyi Sun, Weining Wang +3 more

cs.CV · 29 Jun 2021 · cited 14× · arXiv ↗ · more like this ·

Face sketch synthesis has made significant progress with the development of deep neural networks in these years. The delicate depiction of sketch portraits facilitates a wide range of applications like digital entertainment and law enforcement. However, accurate and realistic face sketch generation is still a challenging task due to the illumination variations and complex backgrounds in the real scenes. To tackle these challenges, we propose a novel Semantic-Driven Generative Adversarial Network (SDGAN) which embeds global structure-level style injection and local class-level knowledge re-weighting. Specifically, we conduct facial saliency detection on the input face photos to provide overall facial texture structure, which could be used as a global type of prior information. In addition, we exploit face parsing layouts as the semantic-level spatial prior to enforce globally structural…

AutoCaption: Image Captioning with Neural Architecture Search

Xinxin Zhu, Weining Wang, Longteng Guo +1 more

cs.CV · December 2020 · cited 9× · arXiv ↗ · more like this ·

Image captioning transforms complex visual information into abstract natural language for representation, which can help computers understanding the world quickly. However, due to the complexity of the real environment, it needs to identify key objects and realize their connections, and further generate natural language. The whole process involves a visual understanding module and a language generation module, which brings more challenges to the design of deep neural networks than other tasks. Neural Architecture Search (NAS) has shown its important role in a variety of image recognition tasks. Besides, RNN plays an essential role in the image captioning task. We introduce a AutoCaption method to better design the decoder module of the image captioning where we use the NAS to design the decoder module called AutoRNN automatically. We use the reinforcement learning method based on shared…

A Survey of Image Emotional Semantic Research

Weining Wang

Computer Vision and Pattern Recognition · Journal of Circuits and Systems · 2003 · cited 9× · OpenAlex ↗ · more like this ·

Human perception and understanding of image are subjective. The motivation of image emotional semantic research is to endow the computer the ability for understanding image in emotional way as human being does. The results are to realize image subjective emotional semantic description, these results can be used in fields such as image emotional evaluation, image retrieval, design support, etc. It is a promising and challenging direction for research. In this paper, status quo of image emotional semantic research is reviewed. Typical system architectures are given, key techniques of the system are analyzed and discussed from three aspects: approaches to image feature extraction, schemes of human emotion description, approaches to build the mapping between image features and emotional space.

Temporal Action Proposal Generation With Action Frequency Adaptive Network

Yepeng Tang, Weining Wang, Chunjie Zhang +2 more

Computer Vision and Pattern Recognition · IEEE Transactions on Multimedia · 13 Jul 2023 · cited 8× · DOI ↗ · more like this ·

As the cornerstone of human-behavior analysis in video understanding, temporal action proposal generation aims to predict the starting and ending time of human action instances in untrimmed videos. Although large achievements in temporal action proposal generation have been achieved, most previous studies ignore the variability of action frequency in raw videos, leading to unsatisfying performances on high-action-frequency videos. In fact, there exists two main issues which should be well addressed: data imbalance between high and low action-frequency videos, and inferior detection of short actions in high-action-frequency videos. To address the above issues, we propose an effective framework by adapting to the variability of action frequency, namely Action Frequency Adaptive Network (AFAN), which can be flexibly built upon any temporal action proposal generation method. AFAN consists…

Semi-Supervised Temporal Action Proposal Generation via Exploiting 2-D Proposal Map

Weining Wang, Tianwei Lin, Dongliang He +4 more

Computer Vision and Pattern Recognition · IEEE Transactions on Multimedia · 13 Aug 2021 · cited 8× · DOI ↗ · more like this ·

Temporal action proposal generation aims to generate temporal video segments containing human actions in untrimmed videos, which is always a preliminary for such video understanding tasks as action localization and temporally description grounding, etc. Fully-supervised solutions, though proven to be effective, suffer much from heavy data annotation overhead. To address this problem, this paper focuses on a rarely investigated yet practical problem of semi-supervised learning for temporal action proposal generation. Firstly, we propose a Proposal Map oriented Mean-Teacher (PM-MT) model, which can use both labeled and unlabeled data for end-to-end model training. Secondly, a Suppression-and-Re-Generation (SRG) strategy is designed to generate high-quality pseudo labels for unlabeled data, which are then used to finetune the model. Extensive experiments demonstrate…

ED-T2V: An Efficient Training Framework for Diffusion-based Text-to-Video Generation

Jiawei Liu, Weining Wang, Wei Liu +2 more

Computer Vision and Pattern Recognition · 18 Jun 2023 · cited 7× · DOI ↗ · more like this ·

Diffusion models have achieved remarkable performance on image generation. However, It is difficult to reproduce this success on video generation because of expensive training cost. In fact, pretrained image generation models have already acquired visual generation capabilities and could be utilized for video generation. Thus, we propose an Efficient training framework for Diffusion-based Text-to-Video generation (ED-T2V), which is built on a pretrained text-to-image generation model. To model the temporal dynamic information, we propose temporal transformer blocks with novel identity attention and temporal cross-attention. ED-T2V has the following advantages: 1) most of the parameters of pretrained model are frozen to inherit the generation capabilities and reduce the training cost; 2) the identity attention requires the currently generated frame to attend to all positions of its…

RETRACTED: Analysis on the Integration of Media Literacy into Ideological and Political Education Based on Computer New Media

Weining Wang

Education · Journal of Physics Conference Series · October 2020 · cited 7× · DOI ↗ · more like this ·

Abstract The era of computer new media brings opportunities to the current ideological and political education in colleges, but also brings serious impact. The constraints of the existing teaching mode make it difficult to improve the teaching result, so it is urgent to improve students’ media literacy. Based on this, this paper first analyses the impact of ideological & political higher education under the context of computer new media, and then studies the current situation and problems of college students’ media literacy in the new media era, and the necessity of improving students’ media literacy based on computer. Finally, the paper puts forward the strategy of integrating the new media literacy of computer into the ideological & political higher teaching.

CAS-AIR-3D Face: A Low-Quality, Multi-Modal and Multi-Pose 3D Face Database

Qi Li, Xiaoxiao Dong, Weining Wang +1 more

Computer Vision and Pattern Recognition · 20 Jul 2021 · cited 6× · DOI ↗ · more like this ·

Benefiting from deep learning with large scale face databases, 2D face recognition has made significant progress in recent years. However, it still highly depends on lighting conditions and human poses, and suffers from face spoofing problem. In contrast, 3D face recognition reveals a new path that can overcome the previous limitations of 2D face recognition. One of the most important problems for 3D face recognition is to construct a suitable database, which can be exploited to train different 3D face recognition algorithms. In this work, we propose a new database, CAS-AIR-3D Face, for low-quality 3D face recognition. It includes 24713 videos from 3093 individuals, which is captured by Intel RealSense SR305. The database contains three modalities: color, depth and near infrared, and is rich in pose, expression, occlusion and distance variations. To the best of our konwledge, CAS-AIR-3D…

Hierarchical Contrastive Learning for Semantic Segmentation

Jie Jiang, Xingjian He, Weining Wang +2 more

Artificial Intelligence · IEEE Transactions on Neural Networks and Learning Systems · 19 Nov 2024 · cited 4× · DOI ↗ · more like this ·

Recently, pixel-to-pixel contrastive learning in single-scale feature space has been widely studied in semantic segmentation to learn a unified feature expression for pixels of the same category. However, the unified representation is too extreme, and the receptive field of each single-scale pixel is limited, which is insufficient to reflect the representative features of the category. To address these problems, this article extends the single-scale feature space to that of multiscale and proposes a hierarchical contrastive learning (Hi-CL) method to explore pixel-to-component semantic relationships. First, we generate multiscale candidate samples by applying several pooling windows with different sizes on a feature map, where different windows may represent different parts of the objects in the image. Then, we prune the sample set through threshold-based criteria to select appropriate…

Local Quantile Regression

Vladimir Spokoiny, Weining Wang, Wolfgang Karl Härdle

math.ST · 30 Aug 2012 · cited 4× · arXiv ↗ · more like this ·

Quantile regression is a technique to estimate conditional quantile curves. It provides a comprehensive picture of a response contingent on explanatory variables. In a flexible modeling framework, a specific form of the conditional quantile curve is not a priori fixed. % Indeed, the majority of applications do not per se require specific functional forms. This motivates a local parametric rather than a global fixed model fitting approach. A nonparametric smoothing estimator of the conditional quantile curve requires to balance between local curvature and stochastic variability. In this paper, we suggest a local model selection technique that provides an adaptive estimator of the conditional quantile regression curve at each design point. Theoretical results claim that the proposed adaptive procedure performs as good as an oracle which would minimize the local estimation risk for the…

DTCWT-based dynamic texture features for visual speech recognition

Xiaohui Feng, Weining Wang

Signal Processing · November 2008 · cited 4× · DOI ↗ · more like this ·

In this paper, new visual dynamic texture features based on dual tree complex wavelet transform (DTCWT) are proposed to capture important lip motion information for speech recognition. Lip texture features are extracted by DTCWT, for its approximate shift invariance and good directional selectivity. Canberra distances between lip texture features in adjacent frames are used as visual dynamic features. Experiments evaluations on Chinese corpus demonstrate that the performance of visual dynamic texture features is superior to static features in visual speech recognition. Compared to commonly used Canberra distances features, the dynamic texture features lead to about 8% improvement on word accuracy.

New three-dimensional displacement measurement system: modified four-beam moire´ interferometer–Twyman/Green

Weining Wang

Computer Vision and Pattern Recognition · Optical Engineering · 2005 · cited 4× · DOI ↗ · more like this ·

An MMI-T/G system composed of a modified four-beam moire interferometer (MMI) and a Twyman/Green (T/G) interferometer is developed. The MMI-T/G can measure 3-D displacement fringe patterns with a single loading on the specimen. In the MMI-T/G system, one incident laser beam is diffracted into multiple beams by the cross-line grating on the specimen, where the ±1 order diffractive beams are redirected back to the specimen by flat mirrors, then rediffracted by the specimen grating onto the CCD, generating the in-plane displacement fringe pattern; the zero-order diffractive beam interferes with a reference beam of the T/G, generating the out-of-plane displacement fringe pattern. Details of the derivation of the fringe-governing equations are given based on the wavefront interference theory. The fringe-governing equations so obtained are comparable with those derived previously based the…

MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation

Mingzhen Sun, Weining Wang, Yanyuan Qiao +5 more

cs.CV · 2 Oct 2024 · cited 3× · arXiv ↗ · more like this ·

Sounding Video Generation (SVG) is an audio-video joint generation task challenged by high-dimensional signal spaces, distinct data formats, and different patterns of content information. To address these issues, we introduce a novel multi-modal latent diffusion model (MM-LDM) for the SVG task. We first unify the representation of audio and video data by converting them into a single or a couple of images. Then, we introduce a hierarchical multi-modal autoencoder that constructs a low-level perceptual latent space for each modality and a shared high-level semantic feature space. The former space is perceptually equivalent to the raw signal space of each modality but drastically reduces signal dimensions. The latter space serves to bridge the information gap between modalities and provides more insightful cross-modal guidance. Our proposed method achieves new state-of-the-art results…

A detection method of lip-smack in spontaneous speech

Yanxiong Li, Qianhua He, Tao Li +1 more

Signal Processing · July 2008 · cited 3× · DOI ↗ · more like this ·

Lip-smack is one of the background events that current speech recognizers can not effectively handle. Little work has been done on the detection of lip-smack. In this paper, lip-smack characteristics in spontaneous speech are analyzed, and then a lip-smack detection method is proposed. A continuous signal is first split into silence and non-silence signal segments, and then potential lip-smack segments are extracted from the non-silence signal segments. Then hidden Markov models (HMMs) are trained and employed to recognize lip-smacks from potential lip-smack segments. The experimental results evaluated on three Mandarin spontaneous speech corpora show that the proposed method can effectively detect lip-smacks in spontaneous speech with an average precision rate and recall rate of 80.9% and 87.4%, respectively.

Learning Knowledge-based Prompts for Robust 3D Mask Presentation Attack Detection

Fangling Jiang, Qi Li, Bing Liu +4 more

cs.CV · May 2025 · cited 2× · arXiv ↗ · more like this ·

3D mask presentation attack detection is crucial for protecting face recognition systems against the rising threat of 3D mask attacks. While most existing methods utilize multimodal features or remote photoplethysmography (rPPG) signals to distinguish between real faces and 3D masks, they face significant challenges, such as the high costs associated with multimodal sensors and limited generalization ability. Detection-related text descriptions offer concise, universal information and are cost-effective to obtain. However, the potential of vision-language multimodal features for 3D mask presentation attack detection remains unexplored. In this paper, we propose a novel knowledge-based prompt learning framework to explore the strong generalization capability of vision-language models for 3D mask presentation attack detection. Specifically, our approach incorporates entities and triples…