arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Masato Mimura

99 papers · 1998–2026

1998: 11999: 02000: 02001: 12002: 12003: 02004: 02005: 02006: 02007: 02008: 02009: 52010: 62011: 22012: 22013: 22014: 42015: 82016: 42017: 42018: 72019: 62020: 92021: 82022: 52023: 32024: 72025: 62026: 8
1998peak 9 in 20202026

Frequent co-authors

Papers

showing 1–25 of 99 RSS

Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization

Yoshiaki Bando, Masato Mimura, Katsutoshi Itoyama +2 more

cs.SD · · cited 133× · arXiv ↗ · more like this ·

This paper presents a statistical method of single-channel speech enhancement that uses a variational autoencoder (VAE) as a prior distribution on clean speech. A standard approach to speech enhancement is to train a deep neural network (DNN) to take noisy speech as input and output clean speech. Although this supervised approach requires a very large amount of pair data for training, it is not robust against unknown environments. Another approach is to use non-negative matrix factorization (NMF) based on basis spectra trained on clean speech in advance and those adapted to noise on the fly. This semi-supervised approach, however, causes considerable signal distortion in enhanced speech due to the unrealistic assumption that speech spectrograms are linear combinations of the basis spectra. Replacing the poor linear generative model of clean speech in NMF with a VAE---a powerful…

Unsupervised Speech Enhancement Based on Multichannel NMF-Informed Beamforming for Noise-Robust Automatic Speech Recognition

Kazuki Shimada, Yoshiaki Bando, Masato Mimura +3 more

cs.SD · · cited 65× · arXiv ↗ · more like this ·

This paper describes multichannel speech enhancement for improving automatic speech recognition (ASR) in noisy environments. Recently, the minimum variance distortionless response (MVDR) beamforming has widely been used because it works well if the steering vector of speech and the spatial covariance matrix (SCM) of noise are given. To estimating such spatial information, conventional studies take a supervised approach that classifies each time-frequency (TF) bin into noise or speech by training a deep neural network (DNN). The performance of ASR, however, is degraded in an unknown noisy environment. To solve this problem, we take an unsupervised approach that decomposes each TF bin into the sum of speech and noise by using multichannel nonnegative matrix factorization (MNMF). This enables us to accurately estimate the SCMs of speech and noise not from observed noisy mixtures but from…

Distilling the Knowledge of BERT for Sequence-to-Sequence ASR

Hayato Futami, Hirofumi Inaguma, Sei Ueno +3 more

cs.CL · · cited 61× · arXiv ↗ · more like this ·

Attention-based sequence-to-sequence (seq2seq) models have achieved promising results in automatic speech recognition (ASR). However, as these models decode in a left-to-right way, they do not have access to context on the right. We leverage both left and right context by applying BERT as an external language model to seq2seq ASR through knowledge distillation. In our proposed method, BERT generates soft labels to guide the training of seq2seq ASR. Furthermore, we leverage context beyond the current utterance as input to BERT. Experimental evaluations show that our method significantly improves the ASR performance from the seq2seq baseline on the Corpus of Spontaneous Japanese (CSJ). Knowledge distillation from BERT outperforms that from a transformer LM that only looks at left context. We also show the effectiveness of leveraging context beyond the current utterance. Our method…

Leveraging Sequence-to-Sequence Speech Synthesis for Enhancing Acoustic-to-Word Speech Recognition

Masato Mimura, Sei Ueno, Hirofumi Inaguma +2 more

Artificial Intelligence · · cited 50× · DOI ↗ · more like this ·

Encoder-decoder models for acoustic-to-word (A2W) automatic speech recognition (ASR) are attractive for their simplicity of architecture and run-time latency while achieving state-of-the-art performances. However, word-based models commonly suffer from the-of-vocabulary (OOV) word problem. They also cannot leverage text data to improve their language modeling capability. Recently, sequence-to-sequence neural speech synthesis models trainable from corpora have been developed and shown to achieve naturalness com- parable to recorded human speech. In this paper, we explore how we can leverage the current speech synthesis technology to tailor the ASR system for a target domain by preparing only a relevant text corpus. From a set of target domain texts, we generate speech features using a sequence-to-sequence speech synthesizer. These artificial speech features together with real speech…

Acoustic-to-Word Attention-Based Model Complemented with Character-Level CTC-Based Model

Sei Ueno, Hirofumi Inaguma, Masato Mimura +1 more

Artificial Intelligence · · cited 47× · DOI ↗ · more like this ·

This paper addresses end-to-end speech recognition which directly maps acoustic features to a word sequence. The acoustic-to-word model is attractive since it does not require an external language model and an elaborate decoder, resulting in extremely simple and fast decoding. The apparent drawback of this modeling is sparseness of training data, particularly for less frequent words. In this paper, we propose a framework complemented with a character-level model. Joint training of the word-level model with the character-level model enhances the generality of deep learning of feature extraction and classification processes, preventing it from overfitting. Moreover, the character-level model is used to decode out-of-vocabulary (OOV) words that are not covered by the word-level model. Since there are choices of connectionist temporal classification (CTC) and attention-based models in the…

Bayesian Learning of a Language Model from Continuous Speech

Graham Neubig, Masato Mimura, Shinsuke Mori +1 more

Artificial Intelligence · IEICE Transactions on Information and Systems · · cited 39× · DOI ↗ · more like this ·

We propose a novel scheme to learn a language model (LM) for automatic speech recognition (ASR) directly from continuous speech. In the proposed method, we first generate phoneme lattices using an acoustic model with no linguistic constraints, then perform training over these phoneme lattices, simultaneously learning both lexical units and an LM. As a statistical framework for this learning problem, we use non-parametric Bayesian statistics, which make it possible to balance the learned model's complexity (such as the size of the learned vocabulary) and expressive power, and provide a principled learning algorithm through the use of Gibbs sampling. Implementation is performed using weighted finite state transducers (WFSTs), which allow for the simple handling of lattice input. Experimental results on natural, adult-directed speech demonstrate that LMs built using only continuous speech…

Enhancing Monotonic Multihead Attention for Streaming ASR

Hirofumi Inaguma, Masato Mimura, Tatsuya Kawahara

eess.AS · · cited 37× · arXiv ↗ · more like this ·

We investigate a monotonic multihead attention (MMA) by extending hard monotonic attention to Transformer-based automatic speech recognition (ASR) for online streaming applications. For streaming inference, all monotonic attention (MA) heads should learn proper alignments because the next token is not generated until all heads detect the corresponding token boundaries. However, we found not all MA heads learn alignments with a naïve implementation. To encourage every head to learn alignments properly, we propose HeadDrop regularization by masking out a part of heads stochastically during training. Furthermore, we propose to prune redundant heads to improve consensus among heads for boundary detection and prevent delayed token generation caused by such heads. Chunkwise attention on each MA head is extended to the multihead counterpart. Finally, we propose head-synchronous beam search…

Multi-speaker Sequence-to-sequence Speech Synthesis for Data Augmentation in Acoustic-to-word Speech Recognition

Sei Ueno, Masato Mimura, Shinsuke Sakai +1 more

Artificial Intelligence · · cited 28× · DOI ↗ · more like this ·

The acoustic-to-word (A2W) automatic speech recognition (ASR) realizes very fast decoding with a simple architecture and achieves state-of-the-art performance. However, the A2W model suffers from the out-of-vocabulary (OOV) word problem and cannot use text-only data to improve the language modeling capability. Meanwhile, sequence-to-sequence neural speech synthesis has also been developed and achieved naturalness comparable to human speech. We investigate leveraging sequence-to-sequence neural speech synthesis to augment training data for the ASR system in a target domain. While speech synthesis model is usually trained with single speaker data, ASR needs to cover a variety of speakers. In this work, we extend the speech synthesizer so that it can output speech of many speakers. The multi-speaker speech synthesizer is trained with a large corpus in the source domain, then used to…

Learning a language model from continuous speech

Graham Neubig, Masato Mimura, Shinsuke Mori +1 more

Artificial Intelligence · · cited 28× · DOI ↗ · more like this ·

This paper presents a new approach to language model construction, learning a language model not from text, but directly from continuous speech. A phoneme lattice is created using acoustic model scores, and Bayesian techniques are used to robustly learn a language model from this noisy input. A novel sampling technique is devised that allows for the integrated learning of word boundaries and an n-gram language model with no prior linguistic knowledge. The proposed techniques were used to learn a language model directly from continuous, potentially large-vocabulary speech. This language model was able to significantly reduce the ASR phoneme error rate over a separate set of test data, and the proposed lattice processing and lexical acquisition techniques were found to be important factors in this improvement. Index Terms: language acquisition, word segmentation, Pitman-Yor language…

Cross-domain speech recognition using nonparallel corpora with cycle-consistent adversarial networks

Masato Mimura, Shinsuke Sakai, Tatsuya Kawahara

Artificial Intelligence · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · · cited 27× · DOI ↗ · more like this ·

Automatic speech recognition (ASR) systems often does not perform well when it is used in a different acoustic domain from the training time, such as utterances spoken in noisy environments or in different speaking styles. We propose a novel approach to cross-domain speech recognition based on acoustic feature mappings provided by a deep neural network, which is trained using nonparallel speech corpora from two different domains and using no phone labels. For training a target domain acoustic model, we generate “fake” target speech features from the labeleld source domain features using a mapping Gf. We can also generate “fake” source features for testing from the target features using the backward mapping G <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">b</sub> which has been learned simultaneously with G f. The mappings G f and G <sub…

Uyghur morpheme-based language models and ASR

Mijit Ablimit, Graham Neubig, Masato Mimura +3 more

Artificial Intelligence · · cited 27× · DOI ↗ · more like this ·

Uyghur language is an agglutinative language in which words are formed by suffixes attaching to a stem (or root). Because of the explosive nature in vocabulary of the agglutinative languages, several morpheme-based language models are built and experiments are implemented. Morpheme is the smallest meaning bearing unit. In this research, morpheme is referred to any of prefix, stem, or suffix. As a result, a large vocabulary ASR system is built on the basis of Julius system. Several ASR results on language models based on different units (word, morpheme, and syllable) are compared.

Time-domain Speech Enhancement Assisted by Multi-resolution Frequency Encoder and Decoder

Hao Shi, Masato Mimura, Longbiao Wang +2 more

cs.SD · · cited 24× · arXiv ↗ · more like this ·

Time-domain speech enhancement (SE) has recently been intensively investigated. Among recent works, DEMUCS introduces multi-resolution STFT loss to enhance performance. However, some resolutions used for STFT contain non-stationary signals, and it is challenging to learn multi-resolution frequency losses simultaneously with only one output. For better use of multi-resolution frequency information, we supplement multiple spectrograms in different frame lengths into the time-domain encoders. They extract stationary frequency information in both narrowband and wideband. We also adopt multiple decoder outputs, each of which computes its corresponding resolution frequency loss. Experimental results show that (1) it is more effective to fuse stationary frequency features than non-stationary features in the encoder, and (2) the multiple outputs consistent with the frequency loss improve…

ASR Rescoring and Confidence Estimation with ELECTRA

Hayato Futami, Hirofumi Inaguma, Masato Mimura +2 more

cs.CL · · cited 24× · arXiv ↗ · more like this ·

In automatic speech recognition (ASR) rescoring, the hypothesis with the fewest errors should be selected from the n-best list using a language model (LM). However, LMs are usually trained to maximize the likelihood of correct word sequences, not to detect ASR errors. We propose an ASR rescoring method for directly detecting errors with ELECTRA, which is originally a pre-training method for NLP tasks. ELECTRA is pre-trained to predict whether each word is replaced by BERT or not, which can simulate ASR error detection on large text corpora. To make this pre-training closer to ASR error detection, we further propose an extended version of ELECTRA called phone-attentive ELECTRA (P-ELECTRA). In the pre-training of P-ELECTRA, each word is replaced by a phone-to-word conversion model, which leverages phone information to generate acoustically similar words. Since our rescoring method is…

Fixed point properties and second bounded cohomology of universal lattices on Banach spaces

Masato Mimura

Mathematical Physics · · cited 21× · OpenAlex ↗ · more like this ·

Let B be any Lp space for p ∈ (1, ∞) or any Banach space isomorphic to a Hilbert space, and k ≧ 0 be integer. We show that if n ≧ 4, then the universal lattice Γ = SLn(ℤ[x1, . . . , xk]) has property (FB) in the sense of Bader–Furman–Gelander–Monod. Namely, any affine isometric action of Γ on B has a global fixed point. The property of having (FB) for all B above is known to be strictly stronger than Kazhdan's property (T). We also define the following generalization of property (FB) for a group: the boundedness property of all affine quasi-actions on B. We name it property (FFB) and prove that the group Γ above also has this property modulo trivial part. The conclusion above implies that the comparison map in degree two from bounded to ordinary cohomology is injective, provided that the associated linear representation does not contain the trivial representation.

Speech dereverberation using long short-term memory

Masato Mimura, Shinsuke Sakai, Tatsuya Kawahara

Signal Processing · · cited 21× · DOI ↗ · more like this ·

Recently, neural networks have been used for not only phone recognition but also denoising and dereverberation. However, the conventional denoising deep autoencoder (DAE) based on the feed-forward structure is not capable of handling very long speech frames of reverberation. LSTM can be effectively trained to reduce the average error between the enhanced signal and the original clean signal by considering the effect of the long past time frames. In this paper, we demonstrate that considering as long as the maximum reverberation time of the database is effective. Since the effect of reverberation varies depending on the phone-class of the whole speech context, we augment the input of the autoencoder with the phone-class information of the past frames as well as the current frame and call this version of the LSTM autoencoder pLSTM. In the speech recognition experiment using the data set…

Waveform-Domain Speech Enhancement Using Spectrogram Encoding for Robust Speech Recognition

Hao Shi, Masato Mimura, Tatsuya Kawahara

Signal Processing · IEEE/ACM Transactions on Audio Speech and Language Processing · · cited 20× · DOI ↗ · more like this ·

While waveform-domain speech enhancement (SE) has been extensively investigated in recent years and achieves state-of-the-art performance in many datasets, spectrogram-based SE tends to show robust and stable enhancement behavior. In this paper, we propose a waveform-spectrogram hybrid method (WaveSpecEnc) to improve the robustness of waveform-domain SE. WaveSpecEnc refines the corresponding temporal feature map by spectrogram encoding in each encoder layer. Incorporating spectral information provides robust human hearing experience performance. However, it has a minor automatic speech recognition (ASR) improvement. Thus, we improve it for robust ASR by further utilizing spectrogram encoding information (WaveSpecEnc+) to both the SE front-end and ASR back-end. Experimental results using the CHiME-4 dataset show that ASR performance in real evaluation sets is consistently improved with…

On strong property (T) and fixed point properties for Lie groups

Tim de Laat, Masato Mimura, Mikael de la Salle

math.FA · · cited 20× · arXiv ↗ · more like this ·

We consider certain strengthenings of property (T) relative to Banach spaces that are satisfied by high rank Lie groups. Let X be a Banach space for which, for all k, the Banach--Mazur distance to a Hilbert space of all k-dimensional subspaces is bounded above by a power of k strictly less than one half. We prove that every connected simple Lie group of sufficiently large real rank depending on X has strong property (T) of Lafforgue with respect to X. As a consequence, we obtain that every continuous affine isometric action of such a high rank group (or a lattice in such a group) on X has a fixed point. This result corroborates a conjecture of Bader, Furman, Gelander and Monod. For the special linear Lie groups, we also present a more direct approach to fixed point properties, or, more precisely, to the boundedness of quasi-cocycles. Without appealing to strong property (T), we prove…

Data Augmentation for ASR Using TTS Via a Discrete Representation

Sei Ueno, Masato Mimura, Shinsuke Sakai +1 more

Artificial Intelligence · 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · · cited 19× · DOI ↗ · more like this ·

While end-to-end automatic speech recognition (ASR) has achieved high performance, it requires a huge amount of paired speech and transcription data for training. Recently, data augmentation methods have actively been investigated. One method is to use a text-to-speech (TTS) system to gen-erate speech data from text-only data and use the generated speech for data augmentation, but it has been found that the synthesized log Mel-scale filterbank (lmfb) features could have a serious mismatch with the real speech features. In this study, we propose a data augmentation method via a discrete speech representation. The TTS model predicts discrete ID sequences instead of lmfb features, and the ASR also uses the ID sequences as training data. We expect that the use of a discrete representation based on vq-wav2vec not only makes TTS training easier but also mitigates the mismatch with real data.…

Singular limit of a chemotaxis-growth model

Aline Bonami, Dorothea Hilhorst, Elisabeth Logak +1 more

Modeling and Simulation · Advances in Differential Equations · · cited 19× · DOI ↗ · more like this ·

We consider a reaction-diffusion-advection system which is a model for chemotaxis with growth. An appropriate singular limit of this system yields a free-boundary problem where the interface motion depends on the mean curvature and on some nonlocal term. We prove local-in-time existence, uniqueness and regularity for this free-boundary problem and investigate some qualitative properties (lack of monotonicity, loss of convexity). We then establish the convergence of the solution of the reaction-diffusion-advection system to the solution of the free-boundary problem.

Speech Corpus of Ainu Folklore and End-to-end Speech Recognition for Ainu Language

Kohei Matsuura, Sei Ueno, Masato Mimura +2 more

cs.CL · · cited 17× · arXiv ↗ · more like this ·

Ainu is an unwritten language that has been spoken by Ainu people who are one of the ethnic groups in Japan. It is recognized as critically endangered by UNESCO and archiving and documentation of its language heritage is of paramount importance. Although a considerable amount of voice recordings of Ainu folklore has been produced and accumulated to save their culture, only a quite limited parts of them are transcribed so far. Thus, we started a project of automatic speech recognition (ASR) for the Ainu language in order to contribute to the development of annotated language archives. In this paper, we report speech corpus development and the structure and performance of end-to-end ASR for Ainu. We investigated four modeling units (phone, syllable, word piece, and word) and found that the syllable-based model performed best in terms of both word and phone recognition accuracy, which were…

Semi-supervised ensemble DNN acoustic model training

Sheng Li, Xugang Lu, Shinsuke Sakai +2 more

Artificial Intelligence · Tokyo Tech Research Repository (Tokyo Institute of Technology) · · cited 17× · DOI ↗ · more like this ·

It is very important to exploit abundant unlabeled speech for improving the acoustic model training in automatic speech recognition (ASR). Semi-supervised training methods incorporate unlabeled data in addition to labeled data to enhance the model training, but it encounters the error-prone label problem. The ensemble training scheme trains a set of models and combines them to make the model more general and robust, but it has not been applied to the unlabeled data. In this work, we propose an effective semi-supervised training of deep neural network (DNN) acoustic models by incorporating the diversity among the ensemble of models. The resultant model improved the performance in the lecture transcription task. Moreover, the proposed method has also shown a potential for DNN adaptation.

Deep autoencoders augmented with phone-class feature for reverberant speech recognition

Masato Mimura, Shinsuke Sakai, Tatsuya Kawahara

Signal Processing · · cited 17× · DOI ↗ · more like this ·

This paper addresses reverberant speech recognition based on front-end processing using DAE (Deep AutoEncoder) coupled with DNN (Deep Neural Network) acoustic model. DAE can effectively and flexibly learn mapping from corrupted speech to the original clean speech based on the deep learning scheme. While this mapping is conventionally conducted only with the acoustic information, we presume the mapping is also dependent on the phone information. Therefore, we propose a new scheme (pDAE), which augments a phone-class feature to the standard acoustic features as input. Two types of the phone-class feature are investigated. One is the hard recognition result of monophones, and the other is a soft representation derived from the posterior outputs of monophone DNN. In the evaluation on the Reverb Challenge 2014 task, the augmented feature in either type results in a significant improvement…

Unsupervised Beamforming Based on Multichannel Nonnegative Matrix Factorization for Noisy Speech Recognition

Kazuki Shimada, Yoshiaki Bando, Masato Mimura +3 more

Signal Processing · · cited 16× · DOI ↗ · more like this ·

This paper presents unsupervised multichannel speech enhancement for noisy speech recognition. Time-frequency (TF) mask estimation has actively been studied for estimating the steering vectors and spatial covariance matrices of speech and noise used for beamforming. The state-of-the-art approach to mask estimation is to use deep neural networks (DNN s) for classifying the TF bins of observed signals into speech and noise. Such a supervised approach, however, does not work well in an unknown environment. To accurately estimate the spatial covariance matrices in an unsupervised manner, we perform blind source separation (BSS) based on multichannel nonnegative matrix factorization (MNMF) for decomposing each TF bin into the components of speech and the other sources (noise). To clarify a suitable type of beamforming for MNMF, we tested both time- invariant and time-varying versions of the…

Automatic transcription system for meetings of the Japanese national congress

Yuya Akita, Masato Mimura, Tatsuya Kawahara

Artificial Intelligence · · cited 15× · DOI ↗ · more like this ·

This paper presents an automatic speech recognition (ASR) system for assisting meeting record creation of the National Congress of Japan. The system is designed to cope with spontaneous characteristics of meeting speech, as well as a variety of topics and speakers. For acoustic model, minimum phone error (MPE) training is applied with several normalization techniques. For language model, we have proposed statistical style transformation to generate spoken-style N-grams and their statistics. We also introduce statistical modeling of pronunciation variation in spontaneous speech. The ASR system was evaluated on real congressional meetings, and achieved word accuracy of 84%. It is also suggested that the ASR-based transcripts with this accuracy level is usable for editing meeting records. Index Terms: Spontaneous speech recognition, congressional speech, minimum phone error training…