arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Martin Radfar

35 papers · 2002–2025

2002: 12003: 12004: 02005: 12006: 52007: 32008: 02009: 12010: 12011: 12012: 02013: 02014: 02015: 02016: 02017: 02018: 02019: 12020: 32021: 62022: 62023: 32024: 02025: 2
2002peak 6 in 20212025

Frequent co-authors

Papers

showing 1–25 of 35 RSS

Single-Channel Speech Separation Using Soft Mask Filtering

Martin Radfar, Richard M. Dansereau

Signal Processing · IEEE Transactions on Audio Speech and Language Processing · · cited 108× · DOI ↗ · more like this ·

We present an approach for separating two speech signals when only one single recording of their linear mixture is available. For this purpose, we derive a filter, which we call the soft mask filter, using minimum mean square error (MMSE) estimation of the log spectral vectors of sources given the mixture's log spectral vectors. The soft mask filter's parameters are estimated using the mean and variance of the underlying sources which are modeled using the Gaussian composite source modeling (CSM) approach. It is also shown that the binary mask filter which has been empirically and extensively used in single-channel speech separation techniques is, in fact, a simplified form of the soft mask filter. The soft mask filtering technique is compared with the binary mask and Wiener filtering approaches when the input consists of male+male, female+female, and male+female mixtures. The…

Context-Aware Transformer Transducer for Speech Recognition

Feng-Ju Chang, Jing Liu, Martin Radfar +4 more

cs.CL · · cited 103× · arXiv ↗ · more like this ·

End-to-end (E2E) automatic speech recognition (ASR) systems often have difficulty recognizing uncommon words, that appear infrequently in the training data. One promising method, to improve the recognition accuracy on such rare words, is to latch onto personalized/contextual information at inference. In this work, we present a novel context-aware transformer transducer (CATT) network that improves the state-of-the-art transformer-based ASR system by taking advantage of such contextual signals. Specifically, we propose a multi-head attention-based context-biasing network, which is jointly trained with the rest of the ASR sub-networks. We explore different techniques to encode contextual data and to create the final attention context vectors. We also leverage both BLSTM and pretrained BERT based models to encode contextual data and guide the network training. Using an in-house far-field…

End-to-End Multi-Channel Transformer for Speech Recognition

Feng-Ju Chang, Martin Radfar, Athanasios Mouchtaris +2 more

eess.AS · · cited 46× · arXiv ↗ · more like this ·

Transformers are powerful neural architectures that allow integrating different modalities using attention mechanisms. In this paper, we leverage the neural transformer architectures for multi-channel speech recognition systems, where the spectral and spatial information collected from different microphones are integrated using attention layers. Our multi-channel transformer network mainly consists of three parts: channel-wise self attention layers (CSA), cross-channel attention layers (CCA), and multi-channel encoder-decoder attention layers (EDA). The CSA and CCA layers encode the contextual relationship within and between channels and across time, respectively. The channel-attended outputs from CSA and CCA are then fed into the EDA layers to help decode the next token given the preceding ones. The experiments show that in a far-field in-house dataset, our method outperforms the…

Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Filling

Jixuan Wang, Kai Wei, Martin Radfar +2 more

cs.AI · · cited 44× · arXiv ↗ · more like this ·

We propose a novel Transformer encoder-based architecture with syntactical knowledge encoded for intent detection and slot filling. Specifically, we encode syntactic knowledge into the Transformer encoder by jointly training it to predict syntactic parse ancestors and part-of-speech of each token via multi-task learning. Our model is based on self-attention and feed-forward layers and does not require external syntactic information to be available at inference time. Experiments show that on two benchmark datasets, our models with only two Transformer encoder layers achieve state-of-the-art results. Compared to the previously best performed model without pre-training, our models achieve absolute F1 score and accuracy improvement of 1.59% and 0.85% for slot filling and intent detection on the SNIPS dataset, respectively. Our models also achieve absolute F1 score and accuracy improvement…

Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understanding

Bhuvan Agrawal, Markus Müller, Martin Radfar +3 more

eess.AS · · cited 40× · arXiv ↗ · more like this ·

End-to-end (E2E) spoken language understanding (SLU) systems can infer the semantics of a spoken utterance directly from an audio signal. However, training an E2E system remains a challenge, largely due to the scarcity of paired audio-semantics data. In this paper, we treat an E2E system as a multi-modal model, with audio and text functioning as its two modalities, and use a cross-modal latent space (CMLS) architecture, where a shared latent space is learned between the `acoustic' and `text' embeddings. We propose using different multi-modal losses to explicitly guide the acoustic embeddings to be closer to the text embeddings, obtained from a semantically powerful pre-trained BERT model. We train the CMLS model on two publicly available E2E datasets, across different cross-modal losses and show that our proposed triplet loss function achieves the best performance. It achieves a…

Multi-Channel Transformer Transducer for Speech Recognition

Feng-Ju Chang, Martin Radfar, Athanasios Mouchtaris +1 more

eess.AS · · cited 26× · arXiv ↗ · more like this ·

Multi-channel inputs offer several advantages over single-channel, to improve the robustness of on-device speech recognition systems. Recent work on multi-channel transformer, has proposed a way to incorporate such inputs into end-to-end ASR for improved accuracy. However, this approach is characterized by a high computational complexity, which prevents it from being deployed in on-device systems. In this paper, we present a novel speech recognition model, Multi-Channel Transformer Transducer (MCTT), which features end-to-end multi-channel training, low computation cost, and low latency so that it is suitable for streaming decoding in on-device speech recognition. In a far-field in-house dataset, our MCTT outperforms stagewise multi-channel models with transformer-transducer up to 6.01% relative WER improvement (WERR). In addition, MCTT outperforms the multi-channel transformer up to…

Scaled factorial hidden Markov models: A new technique for compensating gain differences in model-based single channel speech separation

Martin Radfar, Willy Wong, Richard M. Dansereau +1 more

Signal Processing · · cited 17× · DOI ↗ · more like this ·

In model-based single channel speech separation, factorial hidden Markov models (FHMM) have been successfully applied to model the mixture signal Y(t) = X(t) + V(t) in terms of trained patterns of the speech signals X(t) and V(t). Nonetheless, when the test signals are scaled versions of the trained patterns (i.e. g <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</sub> X(t) and g <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">v</sub> V(t)), the performance of FHMM degrades significantly. In this paper, we introduce a modification to FHMM, called scaled FHMM, which compensates gain difference. In this technique, first, the scale factors are expressed in terms of the target-to-interference ratio (TIR). Then, an iteration quadratic optimization approach is coupled with FHMM to estimate TIR…

Long-Term Gain Estimation in Model-Based Single Channel Speech Separation

Martin Radfar, Richard M. Dansereau

Signal Processing · · cited 11× · DOI ↗ · more like this ·

Model-based single channel speech separation techniques commonly use trained patterns of the individual speakers to separate the speech signals. In most recent proposed techniques, it is assumed that data used in the train and test phase have the same level of energy, a prerequisite which is hardly met in the real situations. Considering this limitation, we propose a technique which estimates the gain associated with the individual speakers from the mixture and thus obviate the need for this assumption. The basic idea is to express the probability density function (PDF) of the mixture in terms of the individual speakers' PDFs and corresponding gains. Then, those patterns and gains which maximize the mixture's PDF are selected and used to recover the speech signals. Experimental results conducted on a wide variety of mixtures with signal-to-signal ratios ranging from 0 to 18 dB show that…

An FPGA based implementation of G.729

N. Mobini, Bijan Vosoughi Vahdat, Martin Radfar

Computer Vision and Pattern Recognition · · cited 10× · DOI ↗ · more like this ·

The main objective of this article is to present the implementation and simulation of a conjugate structure algebraic code excited linear prediction speech coder (CS-ACELP) based upon ITU-T's G.729 recommendation and to optimize it for real-time implementation on an FPGA. The suggested architecture is characterized by pipelining and parallel operation of functional units, using fixed point two's complement representation for integers. The design was functionally verified by utilizing the ModelSim software package from Mentor Graphics Corporation Company and then synthesized by Xilinx Integrated Software Environment (ISE) 6.1 software. Preliminary results show that the overall system delay is less than 2 ms for each frame.

A Novel Low Complexity VQ-Based Single Channel Speech Separation Technique

Martin Radfar, Richard M. Dansereau, Abolghasem Sayadiyan

Signal Processing · · cited 9× · DOI ↗ · more like this ·

In this paper, a new single channel speech separation technique based on vector quantization (VQ) and the MIXMAX approximation is presented. At the core of this approach are two trained codebooks of the quantized feature vectors of speakers, whereby the main evaluation for separation is performed. The performance of the VQ-based approach is evaluated by applying three separate features: log spectrum, modulated lapped transform (MLT) coefficients, and a fusion of pitch and envelop information. The experiments are conducted in two different scenarios: speaker-dependent and speaker independent. The results show that the log spectrum outperforms the other features for speaker-dependent scenario. However, for the speaker-independent scenario, the best results are obtained from applying the pitch-envelop feature

A NEW ALGORITHM FOR TWO-SPEAKER PITCH TRACKING IN SINGLE CHANNEL PARADIGM

Martin Radfar, Abolghasem Sayadiyan, Richard M. Dansereau

Signal Processing · · cited 9× · DOI ↗ · more like this ·

We describe a new method for extracting the pitch contours of two speakers from their mixture. The technique can be effectively applied to the single channel speech separation problem where common blind source separation approaches fail to handle the problem. The algorithm consists of two stages: a sinusoidal spectrogram based track extractor and a joint minimum mean square harmonic modeling pitch estimator. The pitch contours nominees are first determined using the sinusoidal spectrogram. The harmonic pitch estimator is then used to extract the best two pitch contours. We compare the proposed technique with the two state-of-the-art approaches. Experimental results, conducted on a database consisting of mixed signals with the target to jammer ratios (TJR) ranging from -9 to +9 dB, show that our technique significantly outperforms the two other techniques

Sub-8-bit quantization for on-device speech recognition: a regularization-free approach

Kai Zhen, Martin Radfar, Hieu Duy Nguyen +3 more

cs.SD · · cited 8× · arXiv ↗ · more like this ·

For on-device automatic speech recognition (ASR), quantization aware training (QAT) is ubiquitous to achieve the trade-off between model predictive performance and efficiency. Among existing QAT methods, one major drawback is that the quantization centroids have to be predetermined and fixed. To overcome this limitation, we introduce a regularization-free, "soft-to-hard" compression mechanism with self-adjustable centroids in a mu-Law constrained space, resulting in a simpler yet more versatile quantization scheme, called General Quantizer (GQ). We apply GQ to ASR tasks using Recurrent Neural Network Transducer (RNN-T) and Conformer architectures on both LibriSpeech and de-identified far-field datasets. Without accuracy degradation, GQ can compress both RNN-T and Conformer into sub-8-bit, and for some RNN-T layers, to 1-bit for fast and accurate inference. We observe a 30.73% memory…

FANS: Fusing ASR and NLU for on-device SLU

Martin Radfar, Athanasios Mouchtaris, Siegfried Kunzmann +1 more

cs.CL · · cited 7× · arXiv ↗ · more like this ·

Spoken language understanding (SLU) systems translate voice input commands to semantics which are encoded as an intent and pairs of slot tags and values. Most current SLU systems deploy a cascade of two neural models where the first one maps the input audio to a transcript (ASR) and the second predicts the intent and slots from the transcript (NLU). In this paper, we introduce FANS, a new end-to-end SLU model that fuses an ASR audio encoder to a multi-task NLU decoder to infer the intent, slot tags, and slot values directly from a given input audio, obviating the need for transcription. FANS consists of a shared audio encoder and three decoders, two of which are seq-to-seq decoders that predict non null slot tags and slot values in parallel and in an auto-regressive manner. FANS neural encoder and decoders architectures are flexible which allows us to leverage different combinations of…

End-to-End Neural Transformer Based Spoken Language Understanding

Martin Radfar, Athanasios Mouchtaris, Siegfried Kunzmann

cs.CL · · cited 7× · arXiv ↗ · more like this ·

Spoken language understanding (SLU) refers to the process of inferring the semantic information from audio signals. While the neural transformers consistently deliver the best performance among the state-of-the-art neural architectures in field of natural language processing (NLP), their merits in a closely related field, i.e., spoken language understanding (SLU) have not beed investigated. In this paper, we introduce an end-to-end neural transformer-based SLU model that can predict the variable-length domain, intent, and slots vectors embedded in an audio signal with no intermediate token prediction architecture. This new architecture leverages the self-attention mechanism by which the audio signal is transformed to various sub-subspaces allowing to extract the semantic context implied by an utterance. Our end-to-end transformer SLU predicts the domains, intents and slots in the Fluent…

Compute Cost Amortized Transformer for Streaming ASR

Yi Xie, Jonathan Macoskey, Martin Radfar +5 more

cs.CL · · cited 6× · arXiv ↗ · more like this ·

We present a streaming, Transformer-based end-to-end automatic speech recognition (ASR) architecture which achieves efficient neural inference through compute cost amortization. Our architecture creates sparse computation pathways dynamically at inference time, resulting in selective use of compute resources throughout decoding, enabling significant reductions in compute with minimal impact on accuracy. The fully differentiable architecture is trained end-to-end with an accompanying lightweight arbitrator mechanism operating at the frame-level to make dynamic decisions on each input while a tunable loss function is used to regularize the overall level of compute against predictive performance. We report empirical results from experiments using the compute amortized Transformer-Transducer (T-T) model conducted on LibriSpeech data. Our best model can achieve a 60% compute cost reduction…

A Joint Probabilistic-Deterministic Approach using Source-Filter Modeling of Speech Signal for Single Channel Speech Separation

Martin Radfar, Richard M. Dansereau, Abolghasem Sayadiyan

Signal Processing · · cited 6× · DOI ↗ · more like this ·

In this paper, we present a new technique for separating two speech signals from a single recording. For this purpose, we decompose the speech signal into the excitation signal and the vocal tract function and then estimate the components from the mixed speech using a hybrid model. We first express the probability density function (PDF) of the mixed speech's log spectral vectors in terms of the PDFs of the underlying speech signal's vocal tract functions. Then, the mean vectors of PDFs of the vocal tract functions are obtained using a Maximum Likelihood estimator given the mixed signal. Finally, the estimated vocal tract function along with the extracted pitch values are used to reconstruct estimates of the individual speech signals. We compare our model with both an underdetermined blind source separation and a CASA method. The experimental results show our model outperforms both…

A Joint Identification-Separation Technique for Single Channel Speech Separation

Martin Radfar, Richard M. Dansereau, Abolghasem Sayadiyan

Signal Processing · · cited 5× · DOI ↗ · more like this ·

We present a generalized approach to speaker dependent model-based single channel speech separation techniques in which a priori knowledge of the underlying speakers is used to separate speech signals. For this purpose, we add an identification stage by which we first identify the underlying speakers in the mixture and then use the identified speakers' model to separate speech signals. The proposed technique not only preserves the advantages of model-based speaker dependent single channel speech separation algorithms (i.e. high separability) but also is able to separate the speech signals of an unlimited number of speakers given the speakers' models (i.e. generality). Evaluation results conducted on a database consisting of 100 mixed speech signals with target-to-interference ratios (TIR) ranging -9 dB to +9 dB show significant performance improvements over those techniques which use a…

MPtracker: A new multi-pitch detection and separation algorithm for mixed speech signals

Martin Radfar, Richard M. Dansereau, W.-Y. Chan +1 more

Signal Processing · · cited 4× · DOI ↗ · more like this ·

We present MPtracker, a new algorithm for tracking and separating the pitch frequencies of two speakers from their mixture. The pitch frequencies are detected by introducing a novel spectral distortion optimization which takes into account the sinusoidal modeling of the speech signal. The detected pitch frequencies are grouped, separated, and finally an interpolation method is applied to estimate missing pitch frequencies. We evaluated the performance of the proposed technique on 196 mixtures including 48 male-male, 48 female-female, and 96 male-female mixtures with target-to-interference ratios (TIR) ranging from 0 dB to +18 dB. The results show our simple but effective and fast technique significantly outperforms two widely-used approaches.

Gain estimation in model-based single channel speech separation

Martin Radfar, Willy Wong, W.-Y. Chan +1 more

Signal Processing · · cited 4× · DOI ↗ · more like this ·

In most current model based single channel separation techniques, it is assumed that the recording conditions are identical in the training phase and application phase. In this paper, we consider a general case in which training data and application data have different levels of energy and a technique is proposed to estimate the sources' gains which are required for the separation process. We use the periodogram of the speech signal as the selected feature for separation such that the sources' gains are estimated in terms of normalized periodograms of the sources and the mixture. The proposed technique is compared with a state-of-the-art technique which uses AR modeling of the speech signal and maximum likelihood for estimating gain and separating the sources. Experimental results show that our technique not only outperforms this technique in terms of SNR results and gain estimation…

Multi-task RNN-T with Semantic Decoder for Streamable Spoken Language Understanding

Xuandi Fu, Feng-Ju Chang, Martin Radfar +4 more

cs.CL · · cited 3× · arXiv ↗ · more like this ·

End-to-end Spoken Language Understanding (E2E SLU) has attracted increasing interest due to its advantages of joint optimization and low latency when compared to traditionally cascaded pipelines. Existing E2E SLU models usually follow a two-stage configuration where an Automatic Speech Recognition (ASR) network first predicts a transcript which is then passed to a Natural Language Understanding (NLU) module through an interface to infer semantic labels, such as intent and slot tags. This design, however, does not consider the NLU posterior while making transcript predictions, nor correct the NLU prediction error immediately by considering the previously predicted word-pieces. In addition, the NLU model in the two-stage system is not streamable, as it must wait for the audio segments to complete processing, which ultimately impacts the latency of the SLU system. In this work, we propose…

Single Channel Speech Separation using Minimum Mean Square Error Estimation of Sources' Log Spectra

Martin Radfar, Richard M. Dansereau

Signal Processing · · cited 3× · DOI ↗ · more like this ·

We present an approach for separating two speech signals when only one single recording of their linear mixture is available. The log spectra of the sources are estimated from the mixture's log spectrum using minimum mean square error (MMSE) approach. The estimation is obtained from the assumption that the sources are modelled using a set of Gaussian subsources which are related to the mixture using MIXMAX approximation. The resulting estimator has a closed form and is expressed using the mean and variance of Gaussian subsources. In order to obtain the two most likely subsources which generate the mixture, we use the estimation-detection technique. We also show that the binary mask filtering which has been empirically - and with no mathematical justification - used in speech separation techniques is, in fact, a simplified form of the MMSE estimator. The proposed technique is compared…

On the Choice of Window Size in Model-Based Single Channel Speech Separation

Martin Radfar, Richard M. Dansereau, Abolghasem Sayadiyan

Signal Processing · · cited 3× · DOI ↗ · more like this ·

In this paper, we study the effect of window size on the performance of model-based single channel speech separation techniques. The separation system consists of two trained codebooks of the log spectral vectors of each speaker where by the main process of separation is performed. Four Hamming windows with the durations of 20, 32, 64, and 128 msec are used for short-time processing of the data. The experimental results show, in contrast with other speech processing applications, that a longer window must be applied for the separation purpose. Improvements in SNR of 2-3 dB are reported by using an optimal window selection which can significantly improve the performance of current model-based speech separation techniques

Performance analysis of a non-homogeneous GPS system under Markov modulated fluid traffic

Siavash Khorsandi, Martin Radfar

Computer Networks and Communications · · cited 3× · DOI ↗ · more like this ·

This paper presents a near-exact analysis of the generalized processor sharing (GPS) scheduler under Markov modulated fluid sources (MMFS). Due to the complexity of the GPS system, all previous works are based on so called bounding approach providing upper bound using deterministic analysis or decomposition techniques. These bounds are in general quite loose and result in overly conservative resource provisioning. In this paper, a novel approach is used to obtain near-exact solutions for the queue length distribution of each session. The analysis applies to the general case of N parallel queues with multi-class non-homogeneous sources and non-rate-proportional weight setting of sessions. Numerical results for traffic engineering under various system and traffic parameters are reported.

A Neural Prosody Encoder for End-to-End Dialogue Act Classification

Kai Wei, Dillon Knox, Martin Radfar +6 more

Artificial Intelligence · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · · cited 2× · DOI ↗ · more like this ·

Dialogue act classification (DAC) is a critical task for spoken language understanding in dialogue systems. Prosodic features such as energy and pitch have been shown to be useful for DAC. Despite their importance, little research has explored neural approaches to integrate prosodic features into end-to-end (E2E) DAC models which infer dialogue acts directly from audio signals. In this work, we propose an E2E neural architecture that takes into account the need for characterizing prosodic phenomena co-occurring at different levels inside an utterance. A novel part of this architecture is a learnable gating mechanism that assesses the importance of prosodic features and selectively retains core information necessary for E2E DAC. Our proposed model improves DAC accuracy by 1.07% absolute across three publicly available benchmark datasets.

End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoders

Jixuan Wang, Martin Radfar, Kai Wei +1 more

cs.CL · · cited 1× · arXiv ↗ · more like this ·

It is challenging to extract semantic meanings directly from audio signals in spoken language understanding (SLU), due to the lack of textual information. Popular end-to-end (E2E) SLU models utilize sequence-to-sequence automatic speech recognition (ASR) models to extract textual embeddings as input to infer semantics, which, however, require computationally expensive auto-regressive decoding. In this work, we leverage self-supervised acoustic encoders fine-tuned with Connectionist Temporal Classification (CTC) to extract textual embeddings and use joint CTC and SLU losses for utterance-level SLU tasks. Experiments show that our model achieves 4% absolute improvement over the the state-of-the-art (SOTA) dialogue act classification model on the DSTC2 dataset and 1.3% absolute improvement over the SOTA SLU model on the SLURP dataset.