Applications of deep learning to speech enhancement.

Abstract

Deep neural networks (DNNs) have been successfully employed in a broad range of applications,
\nhaving achieved state-of-the-art results in tasks such as acoustic modeling for automatic speech
\nrecognition and image classification. However, their application to speech enhancement problems
\nsuch as denoising, dereverberation, and source separation is more recent work and therefore in more
\npreliminary stages. In this work, we explore DNN-based speech enhancement from three different
\nand complementary points of view.
\nFirst, we propose a model to perform speech dereverberation by estimating its spectral magnitude
\nfrom the reverberant counterpart. Our models are capable of extracting features that take
\ninto account both short and long-term dependencies in the signal through a convolutional encoder
\nand a recurrent neural network for extracting long-term information. Our model outperforms a
\nrecently proposed model that uses different context information depending on the reverberation
\ntime, without requiring any sort of additional input, yielding improvements of up to 0.4 on PESQ,
\n0.3 on STOI, and 1.0 on POLQA relative to reverberant speech. We also show our model is able to
\ngeneralize to real room impulse responses even when only trained with simulated room impulse responses,
\ndifferent speakers, and high reverberation times. Lastly, listening tests show the proposed
\nmethod outperforming benchmark models in reduction of perceived reverberation.
\nWe also study the role of residual and highway connections in deep neural networks for speech
\nenhancement, and verify whether they function in the same way that their digital signal processing
\ncounterparts. We visualize the outputs of such connections, projected back to the spectral domain,
\nin models trained for speech denoising, and show that while skip connections do not necessarily
\nimprove performance with regards to the number of parameters, they make speech enhancement
\nmodels more interpretable. We also discover, through visualization of the hidden units of the
\ncontext-aware model we proposed, that many of the neurons are in a state that we call "stuck" (i.e.
\ntheir outputs is a constant value different from zero). We propose a method to prune those neurons
\naway from the model without having an impact in performance, and compare this method to other
\nmethods in the literature. The proposed method can be applied post hoc to any pretrained models
\nthat use sigmoid or hyperbolic tangent activations. It also leads to dense models, therefore not
\nrequiring special software/hardware to take advantage of the compressed model.
\nFinally, in order to investigate how useful current objective speech quality metrics are for DNNbased
\nspeech enhancement, we performed online listening tests using the outputs of three different
\nDNN-based speech enhancement models for both denoising and dereverberation. When assessing
\nthe predictive power of several objective metrics, we found that existing non-intrusive methods fail
\nat monitoring signal quality. To overcome this limitation, we propose a new metric based on a
\ncombination of a handful of relevant acoustic features. Results inline with those obtained with
\nintrusive measures are then attained. In a leave-one-model-out test, the proposed non-intrusive metric is also shown to outperform two non-intrusive benchmarks for all three DNN enhancement
\nmethods, showing the proposed method is capable of generalizing to unseen models. We then take
\nthe first steps in incorporating such knowledge into the training of DNN-based speech enhancement
\nmodels by designing a quality-aware cost function. While our approach increases PESQ scores for a
\nmodel with a vocoder output, participants of a small-scale preference test considered it less natural
\nthan the sa