arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Runwu Shi

18 papers · 2023–2026

2023: 22024: 22025: 52026: 9
2023peak 9 in 20262026

Frequent co-authors

Papers

showing 1–18 of 18 RSS

Distance Based Single-Channel Target Speech Extraction

Runwu Shi, Benjamin Yen, Kazuhiro Nakadai

eess.AS · · cited 3× · arXiv ↗ · more like this ·

This paper aims to achieve single-channel target speech extraction (TSE) in enclosures by solely utilizing distance information. This is the first work that utilizes only distance cues without using speaker physiological information for single-channel TSE. Inspired by recent single-channel Distance-based separation and extraction methods, we introduce a novel model that efficiently fuses distance information with time-frequency (TF) bins for TSE. Experimental results in both single-room and multi-room scenarios demonstrate the feasibility and effectiveness of our approach. This method can also be employed to estimate the distances of different speakers in mixed speech. Online demos are available at https://runwushi.github.io/distance-demo-page.

The Right Source Is Not Enough: Understanding History Interference in Speech-LLM Memory

HongJin Song, Weiqiao Shan, Runwu Shi +11 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

Speech large language models retain audio history to support multi-turn conversations. Yet remembering an event does not ensure that its information is used only when relevant. We investigate why irrelevant historical audio changes answers even when the queried event remains represented. Controlled experiments on MOSS-Audio and MiMo-Audio show that historical values direct errors toward specific competing answers, beyond a generic effect of context length. Causal interventions distinguish two paths. In open-ended question answering, most history-value errors disappear when subsequent computation cannot directly access the irrelevant source. In multiple-choice answering, many errors persist: early historical influence is carried by question-candidate states even after direct source access is blocked. Low-dimensional interventions further separate source-role information from answer-value…

UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation

Runwu Shi, Kai Li, Yujin Wang +7 more

eess.AS · · cited 0× · arXiv ↗ · more like this ·

Text-to-audio (TTA) generation aims to synthesize realistic audio that faithfully reflects natural-language descriptions. Most TTA systems adopt a two-stage latent paradigm: an audio tokenizer is optimized for reconstruction and then frozen, after which a generative model is trained in the resulting latent space. However, reconstruction-oriented representations may be suboptimal for generation, motivating joint representation and generative learning. To this end, we introduce \textbf{Unite-Audio}, to our knowledge, is the \textbf{first} to jointly learn continuous audio representations and latent flow matching for TTA. By coupling reconstruction with self-supervised generative prediction, Unite-Audio allows the generative objective to directly shape the latent space rather than treating it as a fixed intermediate representation. We further employ Flow-GRPO post-training to improve…

Rethinking Length-Based Training: Batch Composition and Loss Normalization in Speech Token Language Models

Hongjin Song, Runwu Shi, Weiqiao Shan +4 more

cs.CL · · cited 0× · arXiv ↗ · more like this ·

Short-to-long training is a simple curriculum for speech models, but its gains can be difficult to interpret. In speech token language models, length-based training can change the shuffle policy, batch composition, token retention, and token weights under batch-mean loss. We disentangle these factors through matched comparisons. In the tested settings, short-to-long ordering shows no independent benefit when batch composition and token exposure are fixed. First-epoch grouping lowers perplexity for Mimi under batch-mean loss, but this gain is not observed under token-balanced loss. The cross-tokenizer results are consistent with a link between chunk-length variation and token weighting. This work provides a systematic analysis protocol for studying length-based training in variable-length speech models.

Unsupervised Single-Channel Audio Separation with Diffusion Source Priors

Runwu Shi, Chang Li, Jiang Wang +5 more

eess.AS · · cited 0× · arXiv ↗ · more like this ·

Single-channel audio separation aims to separate individual sources from a single-channel mixture. Most existing methods rely on supervised learning with synthetically generated paired data. However, obtaining high-quality paired data in real-world scenarios is often difficult. This data scarcity can degrade model performance under unseen conditions and limit generalization ability. To this end, in this work, we approach this problem from an unsupervised perspective, framing it as a probabilistic inverse problem. Our method requires only diffusion priors trained on individual sources. Separation is then achieved by iteratively guiding an initial state toward the solution through reconstruction guidance. Importantly, we introduce an advanced inverse problem solver specifically designed for separation, which mitigates gradient conflicts caused by interference between the diffusion prior…

Unsupervised Single-Channel Speech Separation with Diffusion under Speaker-Embedding Guidance

Runwu Shi, Kai Li, Yiyan Wang +5 more

eess.AS · · cited 0× · arXiv ↗ · more like this ·

Speech separation is a fundamental task in audio processing, typically addressed with fully supervised systems trained on paired mixtures. While effective, such systems typically rely on synthetic data pipelines, which may not reflect real-world conditions. Instead, we revisit the source-model paradigm, training a diffusion generative model solely on anechoic speech and formulating separation as a diffusion inverse problem. However, unconditional diffusion models lack speaker-level conditioning, they can capture local acoustic structure but produce temporally inconsistent speaker identities in separated sources. To address this limitation, we propose Speaker-Embedding guidance that, during the reverse diffusion process, maintains speaker coherence within each separated track while driving embeddings of different speakers further apart. In addition, we propose a new separation-oriented…

Single-Microphone-Based Sound Source Localization for Mobile Robots in Reverberant Environments

Jiang Wang, Runwu Shi, Benjamin Yen +2 more

cs.RO · · cited 0× · arXiv ↗ · more like this ·

Accurately estimating sound source positions is crucial for robot audition. However, existing sound source localization methods typically rely on a microphone array with at least two spatially preconfigured microphones. This requirement hinders the applicability of microphone-based robot audition systems and technologies. To alleviate these challenges, we propose an online sound source localization method that uses a single microphone mounted on a mobile robot in reverberant environments. Specifically, we develop a lightweight neural network model with only 43k parameters to perform real-time distance estimation by extracting temporal information from reverberant signals. The estimated distances are then processed using an extended Kalman filter to achieve online sound source localization. To the best of our knowledge, this is the first work to achieve online sound source localization…

Single-Channel Target Speech Extraction Utilizing Distance and Room Clues

Runwu Shi, Zirui Lin, Benjamin Yen +3 more

eess.AS · · cited 0× · arXiv ↗ · more like this ·

This paper aims to achieve single-channel target speech extraction (TSE) in enclosures utilizing distance clues and room information. Recent works have verified the feasibility of distance clues for the TSE task, which can imply the sound source's direct-to-reverberation ratio (DRR) and thus can be utilized for speech separation and TSE systems. However, such distance clue is significantly influenced by the room's acoustic characteristics, such as dimension and reverberation time, making it challenging for TSE systems that rely solely on distance clues to generalize across a variety of different rooms. To solve this, we suggest providing room environmental information (room dimensions and reverberation time) for distance-based TSE for better generalization capabilities. Especially, we propose a distance and environment-based TSE model in the time-frequency (TF) domain with learnable…

Domain Adaptive Road Perception Network of Autonomous Vehicles

Rui Wang, Shichun Yang, Yuyi Chen +7 more

Artificial Intelligence · · cited 0× · DOI ↗ · more like this ·

Road conditions can significantly influence the safety of autonomous vehicles (AV). Existing sensors of AV are usually less effective in recognizing road conditions during night and inclement weather. While intelligent tire systems, can identify different road conditions accurately and hardly be affected by weather or illumination conditions. However, data collected under different working conditions differ from each other with significant shifts and are hard to collect due to the tremendous field experiments. To this end, we propose a domain-adaptive model that can extract invariant features to different driving speeds. The proposed method contains two modules: a) a data pre-processing module to extract variable period signals and b) an adversarial transfer learning module for learning invariant features cross working conditions. The field tests have demonstrated that the proposed…

An Efficient Driver Anomaly State Detection Approach Based on End-Cloud Integration and Unsupervised Learning

Jiayi Lu, Yaoguang Cao, Runwu Shi +6 more

Artificial Intelligence · · cited 0× · DOI ↗ · more like this ·

For current autonomous vehicles, real-time monitoring of driver states and prompt identification of abnormal behaviors during the driving process are of paramount importance for safety. This paper proposes an innovative edge-cloud fusion driver anomaly detection system to address the issues of unstable anomaly detection performance and high computational demands of existing driver anomaly detection systems. Our system achieves instantaneous anomaly detection on the edge side while transmitting crucial facial features of the driver to the cloud for long-term anomaly detection. The proposed method employs unsupervised learning techniques to identify challenging and ill-defined abnormal patterns. Through deployment testing on actual hardware platform, our edge-cloud detection system achieved a latency less than 100ms.

Exploring Efficient Waveform Diffusion Models for Foley Sound Generation

Runwu Shi, Chang Li, Jiahui Li +7 more

eess.AS · · arXiv ↗ · more like this ·

Recent advances in diffusion models have enabled high-fidelity Foley sound generation directly in the waveform space. Existing waveform diffusion models primarily rely on time-domain architectures, such as CNN-based U-Nets and DiffWave-style models, or frequency-domain Transformers modeling temporal dependencies. However, these systems are typically built with large model capacities and substantial computational costs, leaving compact and efficient waveform diffusion architectures largely underexplored. In this work, we introduce a Dual-Path (DP) architecture for waveform diffusion that performs dimension-wise self-attention along both subband and frame axes in the time-frequency domain. This DP design enables fine-grained temporal-spectral modeling while maintaining high efficiency. Based on the proposed DP backbone, we develop two variants: DP-DiT and DP-U-Net. Experiments on the…

From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigation

Nabeela Khan, Bowen Wu, Runwu Shi +5 more

cs.RO · · arXiv ↗ · more like this ·

Recent sign language generation (SLG) systems increasingly output dense 3D body representations, which better preserve full-body kinematics and geometry for downstream embodiment on humanoid robots. However, these generated motions frequently exhibit self-intersections such as hand-hand and hand-torso penetration. While such artifacts may be tolerated in offline rendering, they become critical in humanoid execution as they lead to infeasible inverse-kinematics (IK) solutions, collisions, and unstable retargeted trajectories. We present a system-level framework that bridges SLG outputs to humanoid joint-space execution via two components. First, we introduce a volumetric SMPL-X collision-mitigation module that projects generated signing motions toward physically plausible configurations while minimally deviating from the original trajectory. Second, we propose a vision-language-guided…

Joint Residual Reweighting for Classifier Free Guidance in Flow-Matching Zero-Shot TTS

Runwu Shi, Yujin Wang, Hongjin Song +1 more

eess.AS · · arXiv ↗ · more like this ·

Classifier-free guidance (CFG) is widely used in flow-matching-based zero-shot text-to-speech (TTS), where generation is typically controlled by two conditions: the target text and a prompt speech signal. Standard CFG strengthens these conditions jointly, while recent branch-selective guidance methods attempt to enhance text or speaker conditioning separately, often leading to a trade-off between text correctness and speaker similarity. In this paper, we revisit the CFG under independently masked text and speech-prompt conditions, and decompose the guidance field into text, speaker, and joint residuals. We show that conventional speaker-selective guidance entangles the speaker residual with the joint residual, which may disturb text-related generation. Based on this observation, we propose joint residual reweighting, which independently controls the speaker and joint residuals within…

What Do Neural Networks Learn for TDOA Estimation? A Cross-Architecture Probing Study

Yaozhong Kang, Jiang Wang, Runwu Shi +3 more

cs.SD · · arXiv ↗ · more like this ·

Neural networks outperform classical GCC-PHAT for Time-Difference-of-Arrival (TDOA) estimation in noise and reverberation, yet their internal strategy remains unexplored. To uncover it, we turn GCC-PHAT's mathematical steps into diagnostic targets, probing hidden layers of three architectures (MLP, CNN, Transformer) and complementing with gradient attribution and causal frequency masking. We find that cross-power computation consistently emerges across all architectures and conditions, while PHAT whitening, the defining step of GCC-PHAT, fails to emerge. Instead, networks learn a magnitude-aware frequency weighting that preserves per-frequency reliability information discarded by PHAT. This makes PHAT an information bottleneck: removing it from both classical and neural GCC pipelines improves performance under additive noise. On real-world reverberant data, PHAT remains the best…

Fast-SDE: Efficient Single-Microphone Sound Source Distance Estimation in Reverberant Environments

Jiang Wang, Runwu Shi, Yaozhong Kang +3 more

cs.SD · · arXiv ↗ · more like this ·

Sound source distance estimation (SDE) is a critical capability in human-robot interaction. An inappropriate interaction distance not only reduces the reliability of speech acquisition and understanding, but also compromises the naturalness and comfort of the interaction. Most existing SDE methods rely on microphone arrays, however, multi-microphone systems typically require careful hardware synchronization, geometric calibration, and additional space and computational resources, which limits applicability to size-constrained and computability-limited embodied platforms. To alleviate these issues, we propose Fast-SDE, a lightweight single-microphone SDE framework that is suited for deployment on robot platforms with limited computational resources and strict size constraints. Specifically, Fast-SDE employs a subband-based backbone that decomposes the frequency axis into multiple…

Ecologically-Constrained Task Arithmetic for Multi-Taxa Bioacoustic Classifiers Without Shared Data

Ragib Amin Nihal, Benjamin Yen, Runwu Shi +2 more

cs.SD · · arXiv ↗ · more like this ·

Training data for bioacoustics is scattered across taxa, regions, and institutions. Centralizing it all is often infeasible. We show that independently fine-tuned BEATs encoders can be composed into a unified 661-species classifier via task vector arithmetic without sharing data. We find that bioacoustic task vectors are near-orthogonal (cosine 0.01-0.09). Their separation aligns closely with spectral distribution distance, a gradient consistent with the acoustic niche hypothesis. This geometry makes simple averaging optimal while sign-conflict methods reduce accuracy by one to six percentage points. Composition also creates an asymmetric gap: species-rich groups lose accuracy relative to joint training while underrepresented taxa gain, a redistribution useful for equitable biodiversity monitoring. We verify linear mode connectivity across all taxonomic pairs, demonstrate zero-shot…

Weakly Supervised Detection and Temporal Localization of Whale Calls in Long-Duration Bioacoustic Data

Ragib Amin Nihal, Benjamin Yen, Runwu Shi +2 more

cs.SD · · arXiv ↗ · more like this ·

Passive acoustic monitoring (PAM) systems generate continuous recordings spanning months, yet automated bioacoustic analysis of whale calls requires two separate annotation efforts: binary presence labels for classification and precise temporal boundaries for localization. A binary label for a multi-minute recording can be assigned in seconds, but timestamping every call within it requires hours of expert effort. Providing both is infeasible at operational scale. We present DSMIL-LocNet, a weakly supervised multiple instance learning (MIL) framework that performs both classification and temporal localization using only recording-level presence/absence labels. Our dual-stream architecture integrates spectral and temporal features to process recordings of 2--30 minutes without the temporal compression that degrades existing CNN methods on long inputs. On the AcousticTrends BlueFinLibrary…

Bird Vocalization Embedding Extraction Using Self-Supervised Disentangled Representation Learning

Runwu Shi, Katsutoshi Itoyama, Kazuhiro Nakadai

eess.AS · · arXiv ↗ · more like this ·

This paper addresses the extraction of the bird vocalization embedding from the whole song level using disentangled representation learning (DRL). Bird vocalization embeddings are necessary for large-scale bioacoustic tasks, and self-supervised methods such as Variational Autoencoder (VAE) have shown their performance in extracting such low-dimensional embeddings from vocalization segments on the note or syllable level. To extend the processing level to the entire song instead of cutting into segments, this paper regards each vocalization as the generalized and discriminative part and uses two encoders to learn these two parts. The proposed method is evaluated on the Great Tits dataset according to the clustering performance, and the results outperform the compared pre-trained models and vanilla VAE. Finally, this paper analyzes the informative part of the embedding, further compresses…