arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Runwei Ding

39 papers · 2009–2025

2009: 12010: 02011: 02012: 02013: 02014: 32015: 02016: 12017: 12018: 52019: 02020: 72021: 82022: 12023: 102024: 12025: 1
2009peak 10 in 20232025

Frequent co-authors

Papers

showing 1–25 of 39 RSS

Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation

Wenhao Li, Hong Liu, Runwei Ding +3 more

cs.CV · · cited 314× · arXiv ↗ · more like this ·

Despite the great progress in 3D human pose estimation from videos, it is still an open problem to take full advantage of a redundant 2D pose sequence to learn representative representations for generating one 3D pose. To this end, we propose an improved Transformer-based architecture, called Strided Transformer, which simply and effectively lifts a long sequence of 2D joint locations to a single 3D pose. Specifically, a Vanilla Transformer Encoder (VTE) is adopted to model long-range dependencies of 2D pose sequences. To reduce the redundancy of the sequence, fully-connected layers in the feed-forward network of VTE are replaced with strided convolutions to progressively shrink the sequence length and aggregate information from local contexts. The modified VTE is termed as Strided Transformer Encoder (STE), which is built upon the outputs of VTE. STE not only effectively aggregates…

Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition

Tianyu Guo, Hong Liu, Zhan Chen +3 more

cs.CV · · cited 232× · arXiv ↗ · more like this ·

In recent years, self-supervised representation learning for skeleton-based action recognition has been developed with the advance of contrastive learning methods. The existing contrastive learning methods use normal augmentations to construct similar positive samples, which limits the ability to explore novel movement patterns. In this paper, to make better use of the movement patterns introduced by extreme augmentations, a Contrastive Learning framework utilizing Abundant Information Mining for self-supervised action Representation (AimCLR) is proposed. First, the extreme augmentations and the Energy-based Attention-guided Drop Module (EADM) are proposed to obtain diverse positive samples, which bring novel movement patterns to improve the universality of the learned representations. Second, since directly using extreme augmentations may not be able to boost the performance due to the…

A Survey on 3D Skeleton-Based Action Recognition Using Learning Method

Bin Ren, Mengyuan Liu, Runwei Ding +1 more

cs.CV · · cited 226× · arXiv ↗ · more like this ·

3D skeleton-based action recognition (3D SAR) has gained significant attention within the computer vision community, owing to the inherent advantages offered by skeleton data. As a result, a plethora of impressive works, including those based on conventional handcrafted features and learned feature extraction methods, have been conducted over the years. However, prior surveys on action recognition have primarily focused on video or RGB data-dominated approaches, with limited coverage of reviews related to skeleton data. Furthermore, despite the extensive application of deep learning methods in this field, there has been a notable absence of research that provides an introductory or comprehensive review from the perspective of deep learning architectures. To address these limitations, this survey first underscores the importance of action recognition and emphasizes the significance of 3D…

Towards Domain Generalization In Underwater Object Detection

Hong Liu, Pinhao Song, Runwei Ding

Computer Vision and Pattern Recognition · Lirias (KU Leuven) · · cited 96× · DOI ↗ · more like this ·

A General Underwater Object Detector (GUOD) should perform well on most of underwater circumstances. However, with limited underwater dataset, conventional object detection methods suffer from domain shift severely. This paper aims to build a GUOD using small underwater dataset with limited types of water quality. First, we propose a data augmentation method Water Quality Transfer (WQT) to increase domain diversity of the original small dataset. Second, for mining the semantic information from data generated by WQT, Domain Generalization YOLO (DG-YOLO) is proposed, which consists of three parts: YOLOv3, Domain Invariant Module and Invariant Risk Minimization penalty. Finally, experiments on original and synthetic URPC2019 dataset prove that WQT combined with DG-YOLO achieves promising performance of domain generalization in underwater object detection. The source code can be found at…

Edge-guided Representation Learning for Underwater Object Detection

Linhui Dai, Hong Liu, Pinhao Song +3 more

cs.CV · · cited 83× · arXiv ↗ · more like this ·

Underwater object detection (UOD) is crucial for marine economic development, environmental protection, and the planet's sustainable development. The main challenges of this task arise from low-contrast, small objects, and mimicry of aquatic organisms. The key to addressing these challenges is to focus the model on obtaining more discriminative information. We observe that the edges of underwater objects are highly unique and can be distinguished from low-contrast or mimicry environments based on their edges. Motivated by this observation, we propose an Edge-guided Representation Learning Network, termed ERL-Net, that aims to achieve discriminative representation learning and aggregation under the guidance of edge cues. Firstly, we introduce an edge-guided attention module to model the explicit boundary information, which generates more discriminative features. Secondly, a feature…

Cloth Interactive Transformer for Virtual Try-On

Bin Ren, Hao Tang, Fanyang Meng +3 more

cs.CV · · cited 53× · arXiv ↗ · more like this ·

The 2D image-based virtual try-on has aroused increased interest from the multimedia and computer vision fields due to its enormous commercial value. Nevertheless, most existing image-based virtual try-on approaches directly combine the person-identity representation and the in-shop clothing items without taking their mutual correlations into consideration. Moreover, these methods are commonly established on pure convolutional neural networks (CNNs) architectures which are not simple to capture the long-range correlations among the input pixels. As a result, it generally results in inconsistent results. To alleviate these issues, in this paper, we propose a novel two-stage cloth interactive transformer (CIT) method for the virtual try-on task. During the first stage, we design a CIT matching block, aiming to precisely capture the long-range correlations between the cloth-agnostic person…

Weakly-supervised 3D Human Pose Estimation with Cross-view U-shaped Graph Convolutional Network

Guoliang Hua, Hong Liu, Wenhao Li +3 more

cs.CV · · cited 52× · arXiv ↗ · more like this ·

Although monocular 3D human pose estimation methods have made significant progress, it is far from being solved due to the inherent depth ambiguity. Instead, exploiting multi-view information is a practical way to achieve absolute 3D human pose estimation. In this paper, we propose a simple yet effective pipeline for weakly-supervised cross-view 3D human pose estimation. By only using two camera views, our method can achieve state-of-the-art performance in a weakly-supervised manner, requiring no 3D ground truth but only 2D annotations. Specifically, our method contains two steps: triangulation and refinement. First, given the 2D keypoints that can be obtained through any classic 2D detection methods, triangulation is performed across two views to lift the 2D keypoints into coarse 3D poses. Then, a novel cross-view U-shaped graph convolutional network (CV-UGCN), which can explore the…

Spatial-Temporal Data Augmentation Based on LSTM Autoencoder Network for Skeleton-Based Human Action Recognition

Juanhui Tu, Liu Hong, Fanyang Meng +2 more

Computer Vision and Pattern Recognition · · cited 49× · DOI ↗ · more like this ·

Data augmentation is known to be of crucial importance for the generalization of RNN-based methods of skeleton-based human action recognition. Traditional data augmentation methods artificially adopt various transformations merely in spatial domain, which lack effective temporal representation. This paper extends traditional Long Short-Term Memory (LSTM) and presents a novel LSTM autoencoder network (LSTM-AE) for spatial-temporal data augmentation. In the LSTM-AE, the LSTM network preserves the temporal information of skeleton sequences, and the autoencoder architecture can automatically eliminate irrelevant and redundant information. Meanwhile, a regularized cross-entropy loss is defined to guide the LSTM-AE to learn more suitable representations of skeleton data. Experimental results on the currently largest NTU RGB+D dataset and public SmartHome dataset verify that the proposed model…

Co-Evolution of Pose and Mesh for 3D Human Body Estimation from Video

Yingxuan You, Hong Liu, Ti Wang +3 more

cs.CV · · cited 42× · arXiv ↗ · more like this ·

Despite significant progress in single image-based 3D human mesh recovery, accurately and smoothly recovering 3D human motion from a video remains challenging. Existing video-based methods generally recover human mesh by estimating the complex pose and shape parameters from coupled image features, whose high complexity and low representation ability often result in inconsistent pose motion and limited shape patterns. To alleviate this issue, we introduce 3D pose as the intermediary and propose a Pose and Mesh Co-Evolution network (PMCE) that decouples this task into two parts: 1) video-based 3D human pose estimation and 2) mesh vertices regression from the estimated 3D pose and temporal image feature. Specifically, we propose a two-stream encoder that estimates mid-frame 3D pose and extracts a temporal image feature from the input image sequence. In addition, we design a co-evolution…

An Adaptive Method Based on Multiscale Dilated Convolutional Network for Binaural Speech Source Localization

Lulu Wu, Hong Liu, Bing Yang +1 more

Signal Processing · Complexity · · cited 41× · DOI ↗ · more like this ·

Most binaural speech source localization models perform poorly in unprecedentedly noisy and reverberant situations. Here, this issue is approached by modelling a multiscale dilated convolutional neural network (CNN). The time-related crosscorrelation function (CCF) and energy-related interaural level differences (ILD) are preprocessed in separate branches of dilated convolutional network. The multiscale dilated CNN can encode discriminative representations for CCF and ILD, respectively. After encoding, the individual interaural representations are fused to map source direction. Furthermore, in order to improve the parameter adaptation, a novel semiadaptive entropy is proposed to train the network under directional constraints. Experimental results show the proposed method can adaptively locate speech sources in simulated noisy and reverberant environments.

PKU-GoodsAD: A Supermarket Goods Dataset for Unsupervised Anomaly Detection and Segmentation

Jian Zhang, Runwei Ding, Miaoju Ban +1 more

cs.CV · · cited 40× · arXiv ↗ · more like this ·

Visual anomaly detection is essential and commonly used for many tasks in the field of computer vision. Recent anomaly detection datasets mainly focus on industrial automated inspection, medical image analysis and video surveillance. In order to broaden the application and research of anomaly detection in unmanned supermarkets and smart manufacturing, we introduce the supermarket goods anomaly detection (GoodsAD) dataset. It contains 6124 high-resolution images of 484 different appearance goods divided into 6 categories. Each category contains several common different types of anomalies such as deformation, surface damage and opened. Anomalies contain both texture changes and structural changes. It follows the unsupervised setting and only normal (defect-free) images are used for training. Pixel-precise ground truth regions are provided for all anomalies. Moreover, we also conduct a…

A compact representation of human actions by sliding coordinate coding

Runwei Ding, Qianru Sun, Mengyuan Liu +1 more

Computer Vision and Pattern Recognition · International Journal of Advanced Robotic Systems · · cited 37× · DOI ↗ · more like this ·

Human action recognition remains challenging in realistic videos, where scale and viewpoint changes make the problem complicated. Many complex models have been developed to overcome these difficulties, while we explore using low-level features and typical classifiers to achieve the state-of-the-art performance. The baseline model of feature encoding for action recognition is bag-of-words model, which has shown high efficiency but ignores the arrangement of local features. Refined methods compensate for this problem by using a large number of co-occurrence descriptors or a concatenation of the local distributions in designed segments. In contrast, this article proposes to encode the relative position of visual words using a simple but very compact method called sliding coordinates coding (SCC). The SCC vector of each kind of word is only an eight-dimensional vector which is more compact…

WQT and DG-YOLO: towards domain generalization in underwater object detection

Hong Liu, Pinhao Song, Runwei Ding

cs.CV · · cited 34× · arXiv ↗ · more like this ·

A General Underwater Object Detector (GUOD) should perform well on most of underwater circumstances. However, with limited underwater dataset, conventional object detection methods suffer from domain shift severely. This paper aims to build a GUOD with small underwater dataset with limited types of water quality. First, we propose a data augmentation method Water Quality Transfer (WQT) to increase domain diversity of the original small dataset. Second, for mining the semantic information from data generated by WQT, DG-YOLO is proposed, which consists of three parts: YOLOv3, DIM and IRM penalty. Finally, experiments on original and synthetic URPC2019 dataset prove that WQT+DG-YOLO achieves promising performance of domain generalization in underwater object detection.

HTNet: Human Topology Aware Network for 3D Human Pose Estimation

Jialun Cai, Hong Liu, Runwei Ding +3 more

cs.CV · · cited 31× · arXiv ↗ · more like this ·

3D human pose estimation errors would propagate along the human body topology and accumulate at the end joints of limbs. Inspired by the backtracking mechanism in automatic control systems, we design an Intra-Part Constraint module that utilizes the parent nodes as the reference to build topological constraints for end joints at the part level. Further considering the hierarchy of the human topology, joint-level and body-level dependencies are captured via graph convolutional networks and self-attentions, respectively. Based on these designs, we propose a novel Human Topology aware Network (HTNet), which adopts a channel-split progressive strategy to sequentially learn the structural priors of the human topology from multiple semantic levels: joint, part, and body. Extensive experiments show that the proposed method improves the estimation accuracy by 18.7% on the end joints of limbs…

Explore Human Parsing Modality for Action Recognition

Jinfu Liu, Runwei Ding, Yuhang Wen +4 more

cs.CV · · cited 26× · arXiv ↗ · more like this ·

Multimodal-based action recognition methods have achieved high success using pose and RGB modality. However, skeletons sequences lack appearance depiction and RGB images suffer irrelevant noise due to modality limitations. To address this, we introduce human parsing feature map as a novel modality, since it can selectively retain effective semantic features of the body parts, while filtering out most irrelevant noise. We propose a new dual-branch framework called Ensemble Human Parsing and Pose Network (EPP-Net), which is the first to leverage both skeletons and human parsing modalities for action recognition. The first human pose branch feeds robust skeletons in graph convolutional network to model pose features, while the second human parsing branch also leverages depictive parsing feature maps to model parsing festures via convolutional backbones. The two high-level features will be…

Enhancing direct‐path relative transfer function using deep neural network for robust sound source localization

Bing Yang, Runwei Ding, Yutong Ban +2 more

Signal Processing · CAAI Transactions on Intelligence Technology · · cited 26× · DOI ↗ · more like this ·

Abstract This article proposes a deep neural network (DNN)‐based direct‐path relative transfer function (DP‐RTF) enhancement method for robust direction of arrival (DOA) estimation in noisy and reverberant environments. The DP‐RTF refers to the ratio between the direct‐path acoustic transfer functions of the two microphone channels. First, the complex‐value DP‐RTF is decomposed into the inter‐channel intensity difference, and sinusoidal functions of the inter‐channel phase difference in the time‐frequency domain. Then, the decomposed DP‐RTF features from a series of temporal context frames are utilized to train a DNN model, which maps the DP‐RTF features contaminated by noise and reverberation to the clean ones, and meanwhile provides a time‐frequency (TF) weight to indicate the reliability of the mapping. The DP‐RTF enhancement network can help to enhance the DP‐RTF against noise and…

Audio-Visual Keyword Spotting Based on Multidimensional Convolutional Neural Network

Runwei Ding, Cheng Pang, Hong Liu

Signal Processing · · cited 25× · DOI ↗ · more like this ·

The fusion of audio and visual information is one of the most promising solutions for reliable keyword spotting (KWS), particularly when audio is corrupted by noise. KWS aims to detect a specific word in an audio stream, which still remains a challenging problem under noisy environments. In this paper, an audio-visual neural network based on multidimensional convolutional neural network (MCNN) is proposed to perform audio-visual KWS. Firstly, the log mel-spectrogram and lip area sequence are extracted, respectively, from the audio and visual streams, and are taken as the input of the audio-visual neural network. Then, an audio-visual neural network based on MCNN consisting of 2D CNN and 3D CNN is used to model the time-frequency feature of the log mel-spectrogram and the spatiotemporal feature of the lip area sequence, respectively. Finally, the outputs of the audio and visual networks…

Salient pairwise spatio-temporal interest points for real-time activity recognition

Mengyuan Liu, Hong Liu, Qianru Sun +2 more

Computer Vision and Pattern Recognition · CAAI Transactions on Intelligence Technology · · cited 23× · DOI ↗ · more like this ·

Real-time Human action classification in complex scenes has applications in various domains such as visual surveillance, video retrieval and human robot interaction. While, the task is challenging due to computation efficiency, cluttered backgrounds and intro-variability among same type of actions. Spatio-temporal interest point (STIP) based methods have shown promising results to tackle human action classification in complex scenes efficiently. However, the state-of-the-art works typically utilize bag-of-visual words (BoVW) model which only focuses on the word distribution of STIPs and ignore the distinctive character of word structure. In this paper, the distribution of STIPs is organized into a salient directed graph, which reflects salient motions and can be divided into a time salient directed graph and a space salient directed graph, aiming at adding spatio-temporal discriminant…

Learning Explicit Shape and Motion Evolution Maps for Skeleton-Based Human Action Recognition

Hong Liu, Juanhui Tu, Mengyuan Liu +1 more

Computer Vision and Pattern Recognition · · cited 15× · DOI ↗ · more like this ·

Human action recognition based on skeleton sequences has wide applications in human-computer interaction and intelligent surveillance. Although previous methods have successfully applied Long Short-Term Memory(LSTM) networks to model shape evolution of human actions, it still remains a problem to efficiently recognize actions, especially for similar actions from sequential data due to the lack of the details of motion. To solve this problem, this paper presents an improved LSTM-based network to jointly learn explicit long-term shape evolution maps (SEM) and motion evolution maps (MEM). Firstly, human actions are represented as compact SEM and MEM, which mutually compensate. Secondly, these maps are jointly learned by deep LSTM networks to explore high-level temporal dependencies. Then, a weighted aggregate layer (WAL) is designed to aggregate outputs of L-STM networks cross different…

Interweaved Graph and Attention Network for 3D Human Pose Estimation

Ti Wang, Hong Liu, Runwei Ding +3 more

cs.CV · · cited 12× · arXiv ↗ · more like this ·

Despite substantial progress in 3D human pose estimation from a single-view image, prior works rarely explore global and local correlations, leading to insufficient learning of human skeleton representations. To address this issue, we propose a novel Interweaved Graph and Attention Network (IGANet) that allows bidirectional communications between graph convolutional networks (GCNs) and attentions. Specifically, we introduce an IGA module, where attentions are provided with local information from GCNs and GCNs are injected with global information from attentions. Additionally, we design a simple yet effective U-shaped multi-layer perceptron (uMLP), which can capture multi-granularity information for body joints. Extensive experiments on two popular benchmark datasets (i.e. Human3.6M and MPI-INF-3DHP) are conducted to evaluate our proposed method.The results show that IGANet achieves…

GATOR: Graph-Aware Transformer with Motion-Disentangled Regression for Human Mesh Recovery from a 2D Pose

Yingxuan You, Hong Liu, Xia Li +3 more

cs.CV · · cited 8× · arXiv ↗ · more like this ·

3D human mesh recovery from a 2D pose plays an important role in various applications. However, it is hard for existing methods to simultaneously capture the multiple relations during the evolution from skeleton to mesh, including joint-joint, joint-vertex and vertex-vertex relations, which often leads to implausible results. To address this issue, we propose a novel solution, called GATOR, that contains an encoder of Graph-Aware Transformer (GAT) and a decoder with Motion-Disentangled Regression (MDR) to explore these multiple relations. Specifically, GAT combines a GCN and a graph-aware self-attention in parallel to capture physical and hidden joint-joint relations. Furthermore, MDR models joint-vertex and vertex-vertex interactions to explore joint and vertex relations. Based on the clustering characteristics of vertex offset fields, MDR regresses the vertices by composing the…

Grouped Temporal Enhancement Module for Human Action Recognition

Hong Liu, Bin Ren, Mengyuan Liu +1 more

Computer Vision and Pattern Recognition · · cited 8× · DOI ↗ · more like this ·

Temporal information is a significant cue for recognizing human actions from videos. Different from 2D CNN which can only capture spatial information in an efficient way, 3D CNN is good at capturing both spatial and temporal information at the expense of high computational cost. Beyond both methods, this paper presents a Grouped Temporal Enhancement (GTE) module which even outperforms 3D CNN, meanwhile only needs similar low computational cost as 2D CNN. The GTE module firstly decomposes an input video into spatial and temporal groups along channel dimension, and then uses a learnable temporal shift (LTS) operation for efficient temporal modeling. Finally, a 2D convolution filter is used to enhance the ability of LTS for spatial modeling. Extensive experiments on three benchmark datasets validate the effect of our method.

Cross-domain sentiment classification using deep learning approach

Miao Sun, Qi Tan, Runwei Ding +1 more

Computer Vision and Pattern Recognition · · cited 7× · DOI ↗ · more like this ·

Deep learning, as a new unsupervised leaning algorithm, has strong capabilities to learn data representations. Previous work has shown that new features learned by deep learning algorithm help to improve the accuracy of cross-domain classification. In this paper, we firstly propose a modified version of marginalized stacked denoising autoencoders (mSDA). We call it mSDA++ algorithm, which can learn excellent and low-dimensional features for training classifier. In addition, we combine mSDA with EASYADAPT algorithm to further improve the accuracy of cross-domain classification. Then we use SVM, mSDA, mSDA++, and EA+mSDA algorithms to do the cross-domain sentiment classification experiments on Amazon benchmark dataset. The results show that EA+mSDA algorithm attains the best accuracy. Besides, the mSDA++ algorithm can accelerate the subsequent calculation and reduce the data storage space.

Semi‐supervised long short‐term memory for human action recognition

Hong Liu, Chang Liu, Runwei Ding

Computer Vision and Pattern Recognition · The Journal of Engineering · · cited 6× · DOI ↗ · more like this ·

In real human action recognition task, it is a common phenomenon that there are many unlabelled samples and few labelled samples. How to make good use of unlabelled samples to improve the generalisation ability of models is the focus of semi‐supervised learning research. In this study, the authors present two semi‐supervised methods based on long short‐term memory (LSTM) to learn discriminative hidden features. One is the LSTM ladder network, the other is the Symmetrical LSTM network. By them unlabelled samples can be used automatically to improve learning performance without relying on external interaction. Both on the NTU‐RGB + D dataset and the Kinetics dataset, their methods achieve >10 and 5% improvements, separately.

Combining Adaptive Hierarchical Depth Motion Maps With Skeletal Joints for Human Action Recognition

Runwei Ding, Qinqin He, Hong Liu +1 more

Computer Vision and Pattern Recognition · IEEE Access · · cited 6× · DOI ↗ · more like this ·

This paper presents a new framework for human action recognition by fusing human motion with skeletal joints. First, adaptive hierarchical depth motion maps (AH-DMMs) are proposed to capture the shape and motion cues of action sequences. Specifically, AH-DMMs are calculated over adaptive hierarchical windows and Gabor filters are used to encode the texture information of AH-DMMs. Then, spatial distances of skeletal joint positions are computed to characterize the structure information of the human body. Finally, two types of fusion methods including feature-level fusion and decision-level fusion are employed to combine the motion cues and structure information. The experimental results on public benchmark datasets, i.e., MSRAction3D and UTKinect-Action, show the effectiveness of the proposed method.