arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Jiaming Zhou

99 papers · 2012–2026

2012: 22013: 02014: 12015: 02016: 02017: 02018: 02019: 22020: 22021: 32022: 02023: 132024: 242025: 342026: 18
2012peak 34 in 20252026

Frequent co-authors

Papers

showing 1–25 of 99 RSS

Graph-based High-order Relation Modeling for Long-term Action Recognition

Jiaming Zhou, Kun-Yu Lin, Haoxin Li +1 more

Computer Vision and Pattern Recognition · · cited 49× · DOI ↗ · more like this ·

Long-term actions involve many important visual concepts, e.g., objects, motions, and sub-actions, and there are various relations among these concepts, which we call basic relations. These basic relations will jointly affect each other during the temporal evolution of long-term actions, which forms the high-order relations that are essential for long-term action recognition. In this paper, we propose a Graph-based High-order Relation Modeling (GHRM) module to exploit the high-order relations in the long-term actions for long-term action recognition. In GHRM, each basic relation in the long-term actions will be modeled by a graph, where each node represents a segment in a long video. Moreover, when modeling each basic relation, the information from all the other basic relations will be incorporated by GHRM, and thus the high-order relations in the long-term actions can be well…

MusicEval: A Generative Music Dataset with Expert Ratings for Automatic Text-to-Music Evaluation

Cheng Liu, Hui Wang, Jinghua Zhao +6 more

cs.SD · · cited 32× · arXiv ↗ · more like this ·

The technology for generating music from textual descriptions has seen rapid advancements. However, evaluating text-to-music (TTM) systems remains a significant challenge, primarily due to the difficulty of balancing performance and cost with existing objective and subjective evaluation methods. In this paper, we propose an automatic assessment task for TTM models to align with human perception. To address the TTM evaluation challenges posed by the professional requirements of music evaluation and the complexity of the relationship between text and music, we collect MusicEval, the first generative music assessment dataset. This dataset contains 2,748 music clips generated by 31 advanced and widely used models in response to 384 text prompts, along with 13,740 ratings from 14 music experts. Furthermore, we design a CLAP-based assessment model built on this dataset, and our experimental…

Toward Flotation Process Operation-State Identification via Statistical Modeling of Biologically Inspired Gabor Filtering Responses

Jinping Liu, Jiaming Zhou, Zhaohui Tang +5 more

Artificial Intelligence · IEEE Transactions on Cybernetics · · cited 26× · DOI ↗ · more like this ·

This paper presents a froth image statistical modeling-based online flotation process operation-state identification method by introducing a biologically inspired Gabor wavelet transform in accordance with the physiological findings in the biological vision system. It derived the latent probabilistic density models of these biologically inspired Gabor filtering responses (GFRs) based on a versatile intermediate probability modeling frame, Gaussian scale mixture model. It has demonstrated that both the real and the imaginary representation of GFR obey a Laplace distribution. Accordingly, the amplitude representation of GFR obeys a Gamma distribution. Whereas the phase representation of GFR is an important yet frequently ignored aspect in Gabor-based signal analysis; it is demonstrated to be a periodic distribution and can be expressed by a von Mises-like distribution model. Successively…

Human-Centric Transformer for Domain Adaptive Action Recognition

Kun-Yu Lin, Jiaming Zhou, Wei-Shi Zheng

cs.CV · · cited 21× · arXiv ↗ · more like this ·

We study the domain adaptation task for action recognition, namely domain adaptive action recognition, which aims to effectively transfer action recognition power from a label-sufficient source domain to a label-free target domain. Since actions are performed by humans, it is crucial to exploit human cues in videos when recognizing actions across domains. However, existing methods are prone to losing human cues but prefer to exploit the correlation between non-human contexts and associated actions for recognition, and the contexts of interest agnostic to actions would reduce recognition performance in the target domain. To overcome this problem, we focus on uncovering human-centric action cues for domain adaptive action recognition, and our conception is to investigate two aspects of human-centric action cues, namely human cues and human-context interaction cues. Accordingly, our…

Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

Jiaming Zhou, Qihang Zhang, Gangwei Xu +9 more

cs.RO · · cited 20× · arXiv ↗ · more like this ·

Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. To address the scarcity of task-rich paired human-robot data, we propose an automatic pipeline that converts task-sampled robot trajectories into…

kNN-CTC: Enhancing ASR via Retrieval of CTC Pseudo Labels

Jiaming Zhou, Shiwan Zhao, Yaqi Liu +3 more

cs.SD · · cited 19× · arXiv ↗ · more like this ·

The success of retrieval-augmented language models in various natural language processing (NLP) tasks has been constrained in automatic speech recognition (ASR) applications due to challenges in constructing fine-grained audio-text datastores. This paper presents kNN-CTC, a novel approach that overcomes these challenges by leveraging Connectionist Temporal Classification (CTC) pseudo labels to establish frame-level audio-text key-value pairs, circumventing the need for precise ground truth alignments. We further introduce a skip-blank strategy, which strategically ignores CTC blank frames, to reduce datastore size. kNN-CTC incorporates a k-nearest neighbors retrieval mechanism into pre-trained CTC ASR systems, achieving significant improvements in performance. By incorporating a k-nearest neighbors retrieval mechanism into pre-trained CTC ASR systems and leveraging a fine-grained…

Decentralised adaptive output feedback synchronisation tracking control of spacecraft formation flying with time-varying delay

Jiaming Zhou, Gengchen Ma, Youmin Zhang +1 more

Computer Networks and Communications · IET Control Theory and Applications · · cited 18× · DOI ↗ · more like this ·

This study deals with the problem of decentralised cooperative position tracking with time-varying delays of interconnect communication in the leader–follower spacecraft formation flight. More specifically, a class of low-pass linear filters is developed to derive a control law without absolute and relative velocity measurement, which is significant for practical applications with low-cost configurations of spacecrafts. Using a well-chosen Lyapunov function, it proves that the proposed control law can make the spacecrafts in formation flying achieve global convergence to the desired position and velocity relevant to the leader, in the face of modelling uncertainties, external disturbances, as well as non-uniform time-varying delays in communication links. Moreover, to release the heavy communication pressures of the leader, a modified control law is designed by introducing a finite-time…

Useful or not? The discussion of traditional Chinese medicine to treat COVID-19 on a Chinese social networking site

Di Wang, Jiahui Lu, Jiaming Zhou +1 more

Sociology and Political Science · BMJ Global Health · · cited 10× · DOI ↗ · more like this ·

The use of traditional medicine is a global phenomenon, and the WHO advocated its appropriate integration into modern healthcare systems. However, there is a hot debate about the legitimacy of traditional medicine among the general public. Here, we investigated the debate in the Chinese digital context by analysing 1954 responses related to 100 questions about traditional Chinese medicine (TCM) treatment against COVID-19 on the Zhihu platform. Attitude function theory was applied to understand the reasons underlying public attitudes.Results showed that Zhihu users generally held a supportive attitude toward TCM. Their attitudes mainly came from their own experience and traditional media. The general users were more negative while medical professionals were more positive toward TCM. Ego defence (eg, derogating evidence sources) was used the most to support attitudes, followed by value…

MADI: Inter-domain Matching and Intra-domain Discrimination for Cross-domain Speech Recognition

Jiaming Zhou, Shiwan Zhao, Ning Jiang +2 more

cs.CL · · cited 9× · arXiv ↗ · more like this ·

End-to-end automatic speech recognition (ASR) usually suffers from performance degradation when applied to a new domain due to domain shift. Unsupervised domain adaptation (UDA) aims to improve the performance on the unlabeled target domain by transferring knowledge from the source to the target domain. To improve transferability, existing UDA approaches mainly focus on matching the distributions of the source and target domains globally and/or locally, while ignoring the model discriminability. In this paper, we propose a novel UDA approach for ASR via inter-domain MAtching and intra-domain DIscrimination (MADI), which improves the model transferability by fine-grained inter-domain matching and discriminability by intra-domain contrastive discrimination simultaneously. Evaluations on the Libri-Adapt dataset demonstrate the effectiveness of our approach. MADI reduces the relative word…

Enhancing Dysarthric Speech Recognition for Unseen Speakers via Prototype-Based Adaptation

Shiyao Wang, Shiwan Zhao, Jiaming Zhou +2 more

cs.SD · · cited 8× · arXiv ↗ · more like this ·

Dysarthric speech recognition (DSR) presents a formidable challenge due to inherent inter-speaker variability, leading to severe performance degradation when applying DSR models to new dysarthric speakers. Traditional speaker adaptation methodologies typically involve fine-tuning models for each speaker, but this strategy is cost-prohibitive and inconvenient for disabled users, requiring substantial data collection. To address this issue, we introduce a prototype-based approach that markedly improves DSR performance for unseen dysarthric speakers without additional fine-tuning. Our method employs a feature extractor trained with HuBERT to produce per-word prototypes that encapsulate the characteristics of previously unseen speakers. These prototypes serve as the basis for classification. Additionally, we incorporate supervised contrastive learning to refine feature extraction. By…

TwinFormer: Fine-to-Coarse Temporal Modeling for Long-Term Action Recognition

Jiaming Zhou, Kun-Yu Lin, Yu-Kun Qiu +1 more

Computer Vision and Pattern Recognition · IEEE Transactions on Multimedia · · cited 8× · DOI ↗ · more like this ·

The long-term action in untrimmed video generally contains multiple sub-actions, among which various semantic patterns exist ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , the co-occurrence or sequentiality between sub-actions). These semantic patterns are temporally coarse, and correlated with multiple local contexts which encode the local temporal evolution of visual elements ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , hands, objects) in videos. The local contexts and semantic patterns form the inherent fine-to-coarse temporal structure of long-term actions, which is neglected by existing works. Accordingly, in this work we propose TwinFormer, which exploits a novel fine-to-coarse temporal modeling manner to uncover the temporal structure of long-term…

M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper

Jiaming Zhou, Shiwan Zhao, Jiabei He +6 more

cs.SD · · cited 6× · arXiv ↗ · more like this ·

State-of-the-art models like OpenAI's Whisper exhibit strong performance in multilingual automatic speech recognition (ASR), but they still face challenges in accurately recognizing diverse subdialects. In this paper, we propose M2R-whisper, a novel multi-stage and multi-scale retrieval augmentation approach designed to enhance ASR performance in low-resource settings. Building on the principles of in-context learning (ICL) and retrieval-augmented techniques, our method employs sentence-level ICL in the pre-processing stage to harness contextual information, while integrating token-level k-Nearest Neighbors (kNN) retrieval as a post-processing step to further refine the final output distribution. By synergistically combining sentence-level and token-level retrieval strategies, M2R-whisper effectively mitigates various types of recognition errors. Experiments conducted on Mandarin and…

Enhancing Emotion Recognition in Incomplete Data: A Novel Cross-Modal Alignment, Reconstruction, and Refinement Framework

Haoqin Sun, Shiwan Zhao, Shaokai Li +7 more

cs.MM · · cited 6× · arXiv ↗ · more like this ·

Multimodal emotion recognition systems rely heavily on the full availability of modalities, suffering significant performance declines when modal data is incomplete. To tackle this issue, we present the Cross-Modal Alignment, Reconstruction, and Refinement (CM-ARR) framework, an innovative approach that sequentially engages in cross-modal alignment, reconstruction, and refinement phases to handle missing modalities and enhance emotion recognition. This framework utilizes unsupervised distribution-based contrastive learning to align heterogeneous modal distributions, reducing discrepancies and modeling semantic uncertainty effectively. The reconstruction phase applies normalizing flow models to transform these aligned distributions and recover missing modalities. The refinement phase employs supervised point-based contrastive learning to disrupt semantic correlations and accentuate…

RAMPGrasp: Retentive Attention-Based Multiscale Perception Grasp Detection Network

Jianan Huang, Xuebing Liu, Qing Zhu +6 more

Computer Vision and Pattern Recognition · IEEE Transactions on Circuits and Systems for Video Technology · · cited 5× · DOI ↗ · more like this ·

In robotic grasp detection, challenges such as uncertainty in object type, size, and placement within the scene diminish grasping accuracy. However, the inability to effectively locate the graspable area and incomplete feature extraction for grasp detection are two key factors that hinder grasp detection accuracy and are not considered in current methods. This paper presents a novel retentive attention-based multiscale perception grasp detection network (RAMPGrasp) to address this constraint. First, we introduce retentive attention in the feature extraction module, which significantly improves the efficiency of attention score computation for long sequences in visual tasks. Second, we propose a multiscale spatial pyramid attention module, which can effectively adjust the importance of multiscale feature sequences and feature channels, while enhancing the correlation of multiscale…

RAMP+: Retrieval-Augmented MOS Prediction With Prior Knowledge Integration

Hui Wang, Shiwan Zhao, Xiguang Zheng +3 more

Artificial Intelligence · IEEE Transactions on Audio Speech and Language Processing · · cited 4× · DOI ↗ · more like this ·

Neural network-based automatic Mean Opinion Score (MOS) prediction plays a pivotal role in assessing the perceptual quality of synthetic speech. Historically, this field has struggled with limited data resources, a problem not fully resolved by existing methodologies. On the one hand, self-supervised learning (SSL) models could enhance the representational power of the feature extractor by extensive pre-training on large datasets, but fail to address the data scarcity problem affecting the downstream decoder. On the other hand, methods such as data augmentation and the incorporation of individual judges' assessments could improve data utilization, but hard to ensure a satisfactory balance between the scale and quality of the additional dataset. To address these challenges of decoder inefficiency and suboptimal data utilization, we introduce a retrieval-augmented MOS prediction method…

Improving Zero-Shot Chinese-English Code-Switching ASR with kNN-CTC and Gated Monolingual Datastores

Jiaming Zhou, Shiwan Zhao, Hui Wang +4 more

cs.CL · · cited 4× · arXiv ↗ · more like this ·

The kNN-CTC model has proven to be effective for monolingual automatic speech recognition (ASR). However, its direct application to multilingual scenarios like code-switching, presents challenges. Although there is potential for performance improvement, a kNN-CTC model utilizing a single bilingual datastore can inadvertently introduce undesirable noise from the alternative language. To address this, we propose a novel kNN-CTC-based code-switching ASR (CS-ASR) framework that employs dual monolingual datastores and a gated datastore selection mechanism to reduce noise interference. Our method selects the appropriate datastore for decoding each frame, ensuring the injection of language-specific information into the ASR process. We apply this framework to cutting-edge CTC-based models, developing an advanced CS-ASR system. Extensive experiments demonstrate the remarkable effectiveness of…

Diversifying Spatial-Temporal Perception for Video Domain Generalization

Kun-Yu Lin, Jia-Run Du, Yipeng Gao +2 more

cs.CV · · cited 4× · arXiv ↗ · more like this ·

Video domain generalization aims to learn generalizable video classification models for unseen target domains by training in a source domain. A critical challenge of video domain generalization is to defend against the heavy reliance on domain-specific cues extracted from the source domain when recognizing target videos. To this end, we propose to perceive diverse spatial-temporal cues in videos, aiming to discover potential domain-invariant cues in addition to domain-specific cues. We contribute a novel model named Spatial-Temporal Diversification Network (STDN), which improves the diversity from both space and time dimensions of video data. First, our STDN proposes to discover various types of spatial cues within individual frames by spatial grouping. Then, our STDN proposes to explicitly model spatial-temporal dependencies between video contents at multiple space-time scales by…

CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognition

Tian-Hao Zhang, Dinghao Zhou, Guiping Zhong +2 more

cs.SD · · cited 4× · arXiv ↗ · more like this ·

RNN-T models are widely used in ASR, which rely on the RNN-T loss to achieve length alignment between input audio and target sequence. However, the implementation complexity and the alignment-based optimization target of RNN-T loss lead to computational redundancy and a reduced role for predictor network, respectively. In this paper, we propose a novel model named CIF-Transducer (CIF-T) which incorporates the Continuous Integrate-and-Fire (CIF) mechanism with the RNN-T model to achieve efficient alignment. In this way, the RNN-T loss is abandoned, thus bringing a computational reduction and allowing the predictor network a more significant role. We also introduce Funnel-CIF, Context Blocks, Unified Gating and Bilinear Pooling joint network, and auxiliary training strategy to further improve performance. Experiments on the 178-hour AISHELL-1 and 10000-hour WenetSpeech datasets show that…

PB-LRDWWS System for the SLT 2024 Low-Resource Dysarthria Wake-Up Word Spotting Challenge

Shiyao Wang, Jiaming Zhou, Shiwan Zhao +1 more

cs.SD · · cited 3× · arXiv ↗ · more like this ·

For the SLT 2024 Low-Resource Dysarthria Wake-Up Word Spotting (LRDWWS) Challenge, we introduce the PB-LRDWWS system. This system combines a dysarthric speech content feature extractor for prototype construction with a prototype-based classification method. The feature extractor is a fine-tuned HuBERT model obtained through a three-stage fine-tuning process using cross-entropy loss. This fine-tuned HuBERT extracts features from the target dysarthric speaker's enrollment speech to build prototypes. Classification is achieved by calculating the cosine similarity between the HuBERT features of the target dysarthric speaker's evaluation speech and prototypes. Despite its simplicity, our method demonstrates effectiveness through experimental results. Our system achieves second place in the final Test-B of the LRDWWS Challenge.

Self-Prompt Tuning: Enable Autonomous Role-Playing in LLMs

Aobo Kong, Shiwan Zhao, Hao Chen +6 more

cs.CL · · cited 3× · arXiv ↗ · more like this ·

Recent advancements in LLMs have showcased their remarkable role-playing capabilities, able to accurately simulate the dialogue styles and cognitive processes of various roles based on different instructions and contexts. Studies indicate that assigning LLMs the roles of experts, a strategy known as role-play prompting, can enhance their performance in the corresponding domains. However, the prompt needs to be manually designed for the given problem, requiring certain expertise and iterative modifications. To this end, we propose self-prompt tuning, making LLMs themselves generate role-play prompts through fine-tuning. Leveraging the LIMA dataset as our foundational corpus, we employ GPT-4 to annotate role-play prompts for each data points, resulting in the creation of the LIMA-Role dataset. We then fine-tune LLMs like Llama-2-7B and Mistral-7B on LIMA-Role. Consequently, the…

Unsupervised Deep Homography Estimation based on Transformer

Tianjian Jiang, Qiu Fang, Qing Zhu +6 more

Computer Vision and Pattern Recognition · · cited 3× · DOI ↗ · more like this ·

Homography estimation is a crucial problem in computer vision, which aims to provide an optimal transformation matrix for aligning images captured from different viewpoints. Current methods extract shallow features from image pairs and introduce learnable mask modules to improve homography estimation performance. However, they struggle to capture long-term dependencies between features and comprehend the global structures of image features. A deep unsupervised homography learning framework is proposed in this paper, consisting of a weight-sharing feature extraction network and a homography estimation network based on the Transformer model. The former extracts the local features of images, while the latter learns the correlation between them and understands the global features of images, enabling the algorithm to better estimate the homography of unaligned images. Experimental results…

Emotion-Preserving Prosody Anonymization Network for Voice Privacy Protection

Jiabei He, Shiwan Zhao, Jiaming Zhou +3 more

Artificial Intelligence · · cited 2× · DOI ↗ · more like this ·

Balancing emotion preservation and privacy protection in voice anonymization presents a significant challenge, particularly due to the difficulty of effectively handling prosody, a key feature in speech. While preserving prosodic features in anonymized speech enhances emotional expression, it also increases the risk of leaking speaker information. To address this conflict, we propose a lightweight Emotion-Preserving Prosody Anonymization (EPPA) network, which extracts speaker-independent prosodic features to preserve speech emotion while converting them into another speaker’s style for anonymization. By combining EPPA with timbre cloning for anonymization while retaining speech content, we achieve a more balanced voice conversion. Evaluated using the Voice Privacy Challenge (VPC) 2024 metrics, our proposed EPPA, utilizing the closest center distance (CCD) anonymization strategy…

Findings of the 2024 Mandarin Stuttering Event Detection and Automatic Speech Recognition Challenge

Hongfei Xue, Rong Gong, Mingchen Shao +10 more

eess.AS · · cited 2× · arXiv ↗ · more like this ·

The StutteringSpeech Challenge focuses on advancing speech technologies for people who stutter, specifically targeting Stuttering Event Detection (SED) and Automatic Speech Recognition (ASR) in Mandarin. The challenge comprises three tracks: (1) SED, which aims to develop systems for detection of stuttering events; (2) ASR, which focuses on creating robust systems for recognizing stuttered speech; and (3) Research track for innovative approaches utilizing the provided dataset. We utilizes an open-source Mandarin stuttering dataset AS-70, which has been split into new training and test sets for the challenge. This paper presents the dataset, details the challenge tracks, and analyzes the performance of the top systems, highlighting improvements in detection accuracy and reductions in recognition error rates. Our findings underscore the potential of specialized models and augmentation…

Reconstructing the Perceived Faces from Brain Signals without Large Number of Training Samples

H. Zhang, Zhen Wei, Jiaming Zhou +1 more

Computer Vision and Pattern Recognition · PubMed · · cited 2× · DOI ↗ · more like this ·

Reconstructing the perceived faces from brain signals has become a promising work recently. However, the reconstruction accuracies rely on a large number of brain signals collected for training a stable reconstruction model, which is really time consuming, and greatly limits its application. In our current study, we develop a new framework that can efficiently perform high-quality face reconstruction with only a small number of brain signals as training samples. The framework consists of three mathematical models: principle component analysis (PCA), linear regression (LR) and conditional generative adversarial network (cGAN). We conducted a functional Magnetic Resonance Imaging (fMRI) experiment in which two subjects' brain signals were collected to test the efficiency of our proposed method. Results show that we can achieve state-of-the-art reconstruction performance from brain signals…

Multiscale-Hashing Network for 6-D Pose Estimation of Unseen Objects in the Wild

Jiaming Zhou, Qing Zhu, Yaonan Wang +4 more

Computer Vision and Pattern Recognition · IEEE/ASME Transactions on Mechatronics · · cited 1× · DOI ↗ · more like this ·

Estimating the pose of unseen objects is a fundamental task in robotics and industrial automation. Some methods rely on prior knowledge of individual objects for this task and require the model to be trained on specific object instances or categories. Other methods can estimate the pose of unseen objects but are often limited in handling occlusions. To address these challenges, we propose an unseen object pose estimation method for objects not encountered during training. We propose a Hashing Attention Network that integrates features from multiple scales to effectively capture global information while maintaining high sensitivity to local positional details, thereby significantly enhancing the model’s predictive accuracy for occluded objects. We incorporate Locality Sensitive Hashing into the self-attention mechanism of the vision transformer. This approach significantly reduces…