Author

Sining Sun

30 papers · 1753–2025

1966peak 6 in 20182025

Frequent co-authors

Papers

showing 1–25 of 30 RSS

Unsupervised Domain Adaptation via Domain Adversarial Training for Speaker Recognition

Qing Wang, Wei Rao, Sining Sun +3 more

Artificial Intelligence · April 2018 · cited 151× · DOI ↗ · more like this ·

The i-vector approach to speaker recognition has achieved good performance when the domain of the evaluation dataset is similar to that of the training dataset. However, in realworld applications, there is always a mismatch between the training and evaluation datasets, that leads to performance degradation. To address this problem, this paper proposes to learn the domain-invariant and speaker-discriminative speech representations via domain adversarial training. Specifically, with domain adversarial training method, we use a gradient reversal layer to remove the domain variation and project the different domain data into the same subspace. Moreover, we compare the proposed method with other state-of-the-art unsupervised domain adaptation techniques for i-vector approach to speaker recognition (e.g. autoencoder based domain adaptation, inter dataset variability compensation…

Domain Adversarial Training for Accented Speech Recognition

Sining Sun, Ching-Feng Yeh, Mei-Yuh Hwang +2 more

cs.CL · 7 Jun 2018 · cited 149× · arXiv ↗ · more like this ·

In this paper, we propose a domain adversarial training (DAT) algorithm to alleviate the accented speech recognition problem. In order to reduce the mismatch between labeled source domain data ("standard" accent) and unlabeled target domain data (with heavy accents), we augment the learning objective for a Kaldi TDNN network with a domain adversarial training (DAT) objective to encourage the model to learn accent-invariant features. In experiments with three Mandarin accents, we show that DAT yields up to 7.45% relative character error rate reduction when we do not have transcriptions of the accented speech, compared with the baseline trained on standard accent data only. We also find a benefit from DAT when used in combination with training from automatic transcriptions on the accented data. Furthermore, we find that DAT is superior to multi-task learning for accented speech…

Training Augmentation with Adversarial Examples for Robust Speech Recognition

Sining Sun, Ching-Feng Yeh, Mari Ostendorf +2 more

cs.CL · 17 Jun 2018 · cited 65× · arXiv ↗ · more like this ·

This paper explores the use of adversarial examples in training speech recognition systems to increase robustness of deep neural network acoustic models. During training, the fast gradient sign method is used to generate adversarial examples augmenting the original training data. Different from conventional data augmentation based on data transformations, the examples are dynamically generated based on current acoustic model parameters. We assess the impact of adversarial data augmentation in experiments on the Aurora-4 and CHiME-4 single-channel tasks, showing improved robustness against noise and channel variation. Further improvement is obtained when combining adversarial examples with teacher/student training, leading to a 23% relative word error rate reduction on Aurora-4.

An Attention-based Neural Network Approach for Single Channel Speech Enhancement

Xiang Hao, Changhao Shan, Yong Xu +2 more

Signal Processing · 17 Apr 2019 · cited 52× · DOI ↗ · more like this ·

This paper proposes an attention-based neural network approach for single channel speech enhancement. Our work is inspired by the recent success of attention models in sequence-to-sequence learning. It is intuitive to use attention mechanism in speech enhancement as humans are able to focus on the important speech components in an audio stream with "high attention" while perceiving the unimportant region (e.g., noise or interference) in "low attention", and thus adjust the focal point over time. Specifically, taking noisy spectrum as input, our model is composed of an LSTM based encoder, an attention mechanism and a speech generator, resulting in enhanced spectrum. Experiments show that, as compared with OM-LSA and the LSTM baseline, the proposed attention approach can consistently achieve better performance in terms of speech quality (PESQ) and intelligibility (STOI). More promisingly…

Adversarial Examples for Improving End-to-end Attention-based Small-footprint Keyword Spotting

Xiong Wang, Sining Sun, Changhao Shan +4 more

Artificial Intelligence · 17 Apr 2019 · cited 43× · DOI ↗ · more like this ·

In this paper, we explore the use of adversarial examples for improving a neural network based keyword spotting (KWS) system. Specially, in our system, an effective and small-footprint attention-based neural network model is used. Adversarial example is defined as a misclassified example by a model, but it is only slightly skewed from the original correctly-classified one. In the KWS task, it is a natural idea to regard the false alarmed or false rejected queries as some kind of adversarial examples. In our work, given a well-trained attention-based KWS model, we first generate adversarial examples using the fast gradient sign method (FGSM) and find that these examples can dramatically degrade the KWS performance. Using these adversarial examples as augmented data to retrain the KWS model, we finally achieve 45.6% relative and false reject rate (FRR) reduction at 1.0 false alarm rate…

Investigating Generative Adversarial Networks based Speech Dereverberation for Robust Speech Recognition

Ke Wang, Junbo Zhang, Sining Sun +3 more

cs.SD · March 2018 · cited 40× · arXiv ↗ · more like this ·

We investigate the use of generative adversarial networks (GANs) in speech dereverberation for robust speech recognition. GANs have been recently studied for speech enhancement to remove additive noises, but there still lacks of a work to examine their ability in speech dereverberation and the advantages of using GANs have not been fully established. In this paper, we provide deep investigations in the use of GAN-based dereverberation front-end in ASR. First, we study the effectiveness of different dereverberation networks (the generator in GAN) and find that LSTM leads a significant improvement as compared with feed-forward DNN and CNN in our dataset. Second, further adding residual connections in the deep LSTMs can boost the performance as well. Finally, we find that, for the success of GAN, it is important to update the generator and the discriminator using the same mini-batch data…

Tiny Transducer: A Highly-efficient Speech Recognition Model on Edge Devices

Yuekai Zhang, Sining Sun, Long Ma

eess.AS · January 2021 · cited 37× · arXiv ↗ · more like this ·

This paper proposes an extremely lightweight phone-based transducer model with a tiny decoding graph on edge devices. First, a phone synchronous decoding (PSD) algorithm based on blank label skipping is first used to speed up the transducer decoding process. Then, to decrease the deletion errors introduced by the high blank score, a blank label deweighting approach is proposed. To reduce parameters and computation, deep feedforward sequential memory network (DFSMN) layers are used in the transducer encoder, and a CNN-based stateless predictor is adopted. SVD technology compresses the model further. WFST-based decoding graph takes the context-independent (CI) phone posteriors as input and allows us to flexibly bias user-specific information. Finally, with only 0.9M parameters after SVD, our system could give a relative 9.1% - 20.5% improvement compared with a bigger conventional hybrid…

Adversarial Regularization for Attention Based End-to-End Robust Speech Recognition

Sining Sun, Pengcheng Guo, Lei Xie +1 more

Artificial Intelligence · IEEE/ACM Transactions on Audio Speech and Language Processing · 5 Aug 2019 · cited 31× · DOI ↗ · more like this ·

End-to-end speech recognition, such as attention based approaches, is an emerging and attractive topic in recent years. It has achieved comparable performance with the traditional speech recognition framework. Because end-to-end approaches integrate acoustic and linguistic information into one model, the perturbation in the acoustic level such as acoustic noise, could be easily propagated to the linguistic level. Thus improving model robustness in real application environments for these end-to-end systems is crucial. In this paper, in order to make the attention based end-to-end model more robust against noises, we formulate regulation of the objective function with adversarial training examples. Particularly two adversarial regularization techniques, the fast gradient-sign method and the local distributional smoothness method, are explored to improve noise robustness. Experiments on…

Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learning

Songjun Cao, Yueteng Kang, Yanzhe Fu +4 more

eess.AS · 15 Sep 2021 · cited 19× · arXiv ↗ · more like this ·

Recently self-supervised learning has emerged as an effective approach to improve the performance of automatic speech recognition (ASR). Under such a framework, the neural network is usually pre-trained with massive unlabeled data and then fine-tuned with limited labeled data. However, the non-streaming architecture like bidirectional transformer is usually adopted by the neural network to achieve competitive results, which can not be used in streaming scenarios. In this paper, we mainly focus on improving the performance of streaming transformer under the self-supervised learning framework. Specifically, we propose a novel two-stage training method during fine-tuning, which combines knowledge distilling and self-training. The proposed training method achieves 16.3% relative word error rate (WER) reduction on Librispeech noisy test set. Finally, by only using the 100h clean subset of…

Efficient Conformer with Prob-Sparse Attention Mechanism for End-to-End Speech Recognition

Xiong Wang, Sining Sun, Lei Xie +1 more

Artificial Intelligence · 27 Aug 2021 · cited 16× · DOI ↗ · more like this ·

End-to-end models are favored in automatic speech recognition (ASR) because of their simplified system structure and superior performance.Among these models, Transformer and Conformer have achieved state-of-the-art recognition accuracy in which self-attention plays a vital role in capturing important global information.However, the time and memory complexity of self-attention increases squarely with the length of the sentence.In this paper, a prob-sparse self-attention mechanism is introduced into Conformer to sparse the computing process of self-attention in order to accelerate inference speed and reduce space consumption.Specifically, we adopt a Kullback-Leibler divergence based sparsity measurement for each query to decide whether we compute the attention function on this query.By using the prob-sparse attention mechanism, we achieve impressively 8% to 45% inference speed-up and 15%…

CaTT-KWS: A Multi-stage Customized Keyword Spotting Framework based on Cascaded Transducer-Transformer

Zhanheng Yang, Sining Sun, Jin Li +4 more

cs.SD · 4 Jul 2022 · cited 12× · arXiv ↗ · more like this ·

Customized keyword spotting (KWS) has great potential to be deployed on edge devices to achieve hands-free user experience. However, in real applications, false alarm (FA) would be a serious problem for spotting dozens or even hundreds of keywords, which drastically affects user experience. To solve this problem, in this paper, we leverage the recent advances in transducer and transformer based acoustic models and propose a new multi-stage customized KWS framework named Cascaded Transducer-Transformer KWS (CaTT-KWS), which includes a transducer based keyword detector, a frame-level phone predictor based force alignment module and a transformer based decoder. Specifically, the streaming transducer module is used to spot keyword candidates in audio stream. Then force alignment is implemented using the phone posteriors predicted by the phone predictor to finish the first stage keyword…

Conversational Speech Recognition By Learning Conversation-level Characteristics

Kun Wei, Yike Zhang, Sining Sun +2 more

cs.SD · 17 Feb 2022 · cited 12× · arXiv ↗ · more like this ·

Conversational automatic speech recognition (ASR) is a task to recognize conversational speech including multiple speakers. Unlike sentence-level ASR, conversational ASR can naturally take advantages from specific characteristics of conversation, such as role preference and topical coherence. This paper proposes a conversational ASR model which explicitly learns conversation-level characteristics under the prevalent end-to-end neural framework. The highlights of the proposed model are twofold. First, a latent variational module (LVM) is attached to a conformer-based encoder-decoder ASR backbone to learn role preference and topical coherence. Second, a topic model is specifically adopted to bias the outputs of the decoder to words in the predicted topics. Experiments on two Mandarin conversational ASR tasks show that the proposed model achieves a maximum 12% relative character error rate…

Virtual Adversarial Training for DS-CNN Based Small-Footprint Keyword Spotting

Xiong Wang, Sining Sun, Lei Xie

Artificial Intelligence · 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · December 2019 · cited 10× · DOI ↗ · more like this ·

Serving as the tigger of a voice-enabled user interface, on-device keyword spotting model has to be extremely compact, efficient and accurate. In this paper, we adopt a depth-wise separable convolutional neural network (DS-CNN) as our small-footprint KWS model, which is highly competitive to these ends. However, recent study has shown that a compact KWS system is very vulnerable to small adversarial perturbations while augmenting the training data with specifically-generated adversarial examples can improve performance. In this paper, we further improve KWS performance through a virtual adversarial training (VAT) solution. Instead of using adversarial examples for data augmentation, we propose to train a DS-CNN KWS model using adversarial regularization, which aims to smooth model's distribution and thus to improve robustness, by explicitly introducing a distribution smoothness measure…

Domain Adversarial Training for Improving Keyword Spotting Performance of ESL Speech

Jingyong Hou, Pengcheng Guo, Sining Sun +3 more

Artificial Intelligence · 17 Apr 2019 · cited 10× · DOI ↗ · more like this ·

A second language (L2) learner usually cannot speak L2 well in both pronunciations and forming-of-words. Hence his/her L2 speech cannot be well recognized by a recognizer trained with native data. Domain adversarial training (DAT), capable of reducing the acoustic mismatch between training and testing, can be useful for improving speech recognition of L2 learners. To get around the ungrammatical L2 speech in scenario-based conversation training, keyword spotting (KWS) is an effective solution by relaxing the language model constraint in decoding. On the acoustic pronunciation side, DAT is investigated in this study for training a neural net-based acoustic model. DAT model is trained with both native and English as second language (ESL) learners' speech to extract more invariant features from native to ESL speech by equalizing their intrinsic difference. The model is jointly optimized…

Leveraging Acoustic Contextual Representation by Audio-textual Cross-modal Learning for Conversational ASR

Kun Wei, Yike Zhang, Sining Sun +2 more

eess.AS · 3 Jul 2022 · cited 9× · arXiv ↗ · more like this ·

Leveraging context information is an intuitive idea to improve performance on conversational automatic speech recognition(ASR). Previous works usually adopt recognized hypotheses of historical utterances as preceding context, which may bias the current recognized hypothesis due to the inevitable historicalrecognition errors. To avoid this problem, we propose an audio-textual cross-modal representation extractor to learn contextual representations directly from preceding speech. Specifically, it consists of two modal-related encoders, extracting high-level latent features from speech and the corresponding text, and a cross-modal encoder, which aims to learn the correlation between speech and text. We randomly mask some input tokens and input sequences of each modality. Then a token-missing or modal-missing prediction with a modal-level CTC loss on the cross-modal encoder is performed.…

Multi-head Monotonic Chunkwise Attention For Online Speech Recognition

Baiji Liu, Songjun Cao, Sining Sun +2 more

cs.CL · May 2020 · cited 9× · arXiv ↗ · more like this ·

The attention mechanism of the Listen, Attend and Spell (LAS) model requires the whole input sequence to calculate the attention context and thus is not suitable for online speech recognition. To deal with this problem, we propose multi-head monotonic chunk-wise attention (MTH-MoChA), an improved version of MoChA. MTH-MoChA splits the input sequence into small chunks and computes multi-head attentions over the chunks. We also explore useful training strategies such as LSTM pooling, minimum world error rate training and SpecAugment to further improve the performance of MTH-MoChA. Experiments on AISHELL-1 data show that the proposed model, along with the training strategies, improve the character error rate (CER) of MoChA from 8.96% to 7.68% on test set. On another 18000 hours in-car speech data set, MTH-MoChA obtains 7.28% CER, which is significantly better than a state-of-the-art hybrid…

Direction of Arrival Estimation Using Amplitude and Phase Information in Low-Profile MIMO Arrays

Hui Li, Sining Sun, Jie Wang

Signal Processing · IEEE Transactions on Antennas and Propagation · 24 Aug 2018 · cited 7× · DOI ↗ · more like this ·

Accurate direction of arrival (DOA) estimation in compact terminal devices using small antenna arrays is challenging due to their limited radiating apertures. In this communication, we employ both the amplitude and the phase information to improve the estimation accuracy and eliminate phase ambiguity without increasing the size of the antenna array. The optimal amplitude ratio between the antenna patterns is theoretically studied using the theories of the Fisher information, Cramér-Rao bound, and ambiguity function. Criteria of proper amplitude ratio are summarized from the theoretical investigations. Following the guidelines, two antenna arrays are modified to build effective antenna patterns and improve the estimation accuracy. Specifically, the compact monopole-patch array and the dual-monopole array reduce the DOA estimation errors by over 60% and 20%-40%, respectively, under a…

Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction

Zhaoxi Mu, Xinyu Yang, Sining Sun +1 more

cs.SD · December 2023 · cited 5× · arXiv ↗ · more like this ·

Speech signals are inherently complex as they encompass both global acoustic characteristics and local semantic information. However, in the task of target speech extraction, certain elements of global and local semantic information in the reference speech, which are irrelevant to speaker identity, can lead to speaker confusion within the speech extraction network. To overcome this challenge, we propose a self-supervised disentangled representation learning method. Our approach tackles this issue through a two-phase process, utilizing a reference speech encoding network and a global information disentanglement network to gradually disentangle the speaker identity information from other irrelevant factors. We exclusively employ the disentangled speaker identity information to guide the speech extraction network. Moreover, we introduce the adaptive modulation Transformer to ensure that…

Efficient Conformer with Prob-Sparse Attention Mechanism for End-to-EndSpeech Recognition

Xiong Wang, Sining Sun, Lei Xie +1 more

cs.SD · 17 Jun 2021 · cited 5× · arXiv ↗ · more like this ·

End-to-end models are favored in automatic speech recognition (ASR) because of their simplified system structure and superior performance. Among these models, Transformer and Conformer have achieved state-of-the-art recognition accuracy in which self-attention plays a vital role in capturing important global information. However, the time and memory complexity of self-attention increases squarely with the length of the sentence. In this paper, a prob-sparse self-attention mechanism is introduced into Conformer to sparse the computing process of self-attention in order to accelerate inference speed and reduce space consumption. Specifically, we adopt a Kullback-Leibler divergence based sparsity measurement for each query to decide whether we compute the attention function on this query. By using the prob-sparse attention mechanism, we achieve impressively 8% to 45% inference speed-up and…

A Robust Nonlinear Microphone Array Postfilter for Noise Reduction

Suliang Bu, Yunxin Zhao, Mei-Yuh Hwang +1 more

Signal Processing · September 2018 · cited 4× · DOI ↗ · more like this ·

We propose a robust nonlinear microphone array postfilter for noise reduction. This postfilter is formulated as a function of noise power ratio before and after beamforming and a local speech-to-observation power ratio. The two ratios are readily obtained during beamforming, and can be approximated by local speech posterior probability or time-frequency masks of neural network. This avoids the difficulty in estimating local speech and noise variances of a beamformed signal. On the CHiME-3 test set, we have evaluated our proposed postfilter in comparison with two other postfiltering methods, and our proposed postfilter has produced the best objective scores on the simulated noisy speech as well as higher listening preference scores on real noisy speech.

Direction of arrival estimation using compact MIMO array for portbale devices

Sining Sun, Hui Li, Jiang Xiong

Signal Processing · July 2017 · cited 3× · DOI ↗ · more like this ·

Accurate direction of arrival (DOA) estimation attracts more intention for smart devices in wireless communication. However, designing compact antenna array for estimation with high accuracy is challenging. In this work, a closely packed MIMO array is particularly designed for proper amplitude ratio over different angles. Combining with the phase information, the accuracy and robustness of the estimation is improved, with phase ambiguity eliminated as well. Simulation results show that the proposed MIMO array obtains a maximum and averaged RMS error of 2 degrees and less than 1 degree, respectively, over the entire azimuth plane with an SNR of 10 dB. It performs well at both high and low SNR scenarios, and can be easily integrated in devices such as Wi-Fi router.

Key Frame Mechanism For Efficient Conformer Based End-to-end Speech Recognition

Peng Fan, Changhao Shan, Sining Sun +2 more

cs.SD · 28 Oct 2023 · cited 2× · arXiv ↗ · more like this ·

Recently, Conformer as a backbone network for end-to-end automatic speech recognition achieved state-of-the-art performance. The Conformer block leverages a self-attention mechanism to capture global information, along with a convolutional neural network to capture local information, resulting in improved performance. However, the Conformer-based model encounters an issue with the self-attention mechanism, as computational complexity grows quadratically with the length of the input sequence. Inspired by previous Connectionist Temporal Classification (CTC) guided blank skipping during decoding, we introduce intermediate CTC outputs as guidance into the downsampling procedure of the Conformer encoder. We define the frame with non-blank output as key frame. Specifically, we introduce the key frame-based self-attention (KFSA) mechanism, a novel method to reduce the computation of the…

Multiple fixed beamformers with a spacial Wiener-form postfilter for far-field speech recognition

Sining Sun, Shuran Zhou, Mei-Yuh Hwang +3 more

Signal Processing · November 2019 · cited 2× · DOI ↗ · more like this ·

Far-field speech recognition is becoming a hot topic in research and industrial applications. In this paper, in order to improve far-field speech recognition performance, we propose to use multiple fixed beamformers with a spacial Wiener-form postfilter (MFB-SWP) to suppress noise and interference. Our proposed method consists of two parts, beamforming and post-filter estimation. First, multiple fixed beamformers are designed and each of them aims at one specific direction. Next the target speech is estimated using the fixed beamformer aiming to the target direction, and the noise and interference signals are estimated using the remaining beamformers. After that, we calculate a spacial Wiener-form time-frequency and frame-level gains, as postfilter to further reduce the residual noise and interference. Compared with a single fixed beamformer, the proposed MFB-SWP method can suppress…

DQ-Data2vec: Decoupling Quantization for Multilingual Speech Recognition

Qijie Shao, Linhao Dong, Kun Wei +2 more

cs.SD · 23 Jan 2025 · cited 1× · arXiv ↗ · more like this ·

Data2vec is a self-supervised learning (SSL) approach that employs a teacher-student architecture for contextual representation learning via masked prediction, demonstrating remarkable performance in monolingual ASR. Previous studies have revealed that data2vec's shallow layers capture speaker and language information, middle layers encode phoneme and word features, while deep layers are responsible for reconstruction. Language and phoneme features are crucial for multilingual ASR. However, data2vec's masked representation generation relies on multi-layer averaging, inevitably coupling these features. To address this limitation, we propose a decoupling quantization based data2vec (DQ-Data2vec) for multilingual ASR, which includes a data2vec backbone and two improved online K-means quantizers. Our core idea is using the K-means quantizer with specified cluster numbers to decouple…

Streaming Decoder-Only Automatic Speech Recognition with Discrete Speech Units: A Pilot Study

Peikun Chen, Sining Sun, Changhao Shan +2 more

cs.SD · 27 Jun 2024 · cited 0× · arXiv ↗ · more like this ·

Unified speech-text models like SpeechGPT, VioLA, and AudioPaLM have shown impressive performance across various speech-related tasks, especially in Automatic Speech Recognition (ASR). These models typically adopt a unified method to model discrete speech and text tokens, followed by training a decoder-only transformer. However, they are all designed for non-streaming ASR tasks, where the entire speech utterance is needed during decoding. Hence, we introduce a decoder-only model exclusively designed for streaming recognition, incorporating a dedicated boundary token to facilitate streaming recognition and employing causal attention masking during the training phase. Furthermore, we introduce right-chunk attention and various data augmentation techniques to improve the model's contextual modeling abilities. While achieving streaming speech recognition, experiments on the AISHELL-1 and -2…