On the Usefulness of Statistical Normalisation of Bottleneck Features for Speech Recognition
On the Usefulness of Statistical Normalisation of Bottleneck Features for Speech Recognition
复制标题
DOI：
10.1109/icassp.2019.8683330
复制
发表时间：
2019-05
期刊：
影响因子：
--
通讯作者：
Erfan Loweimi;P. Bell;S. Renals
中科院分区：
文献类型：
--
作者：
Erfan Loweimi;P. Bell;S. Renals
DNN在最先进的ASR系统中发挥着重要作用。它们可用于提取特征和构建声学和语言建模的概率模型。尽管它们在实践中取得了巨大的成功，但理论理解的水平仍然很浅。本文从统计的角度研究DNN。特别是，激活函数的预激活和激活的分布的影响进行了调查和讨论，从分析和经验的观点。这项研究表明，瓶颈层中的预激活密度可以很好地与具有几个高斯的对角GMM拟合，以及ReLU激活函数如何以及为什么促进稀疏性。受预激活的统计特性的启发，还研究了瓶颈特征的统计归一化的有用性。为此，采用了均值（方差）归一化、高斯化和直方图均衡化（HEQ）等方法，并在Aurora-4任务中实现了高达2%（绝对）的WER降低。
DNNs play a major role in the state-of-the-art ASR systems. They can be used for extracting features and building probabilistic models for acoustic and language modelling. Despite their huge practical success, the level of theoretical understanding has remained shallow. This paper investigates DNNs from a statistical standpoint. In particular, the effect of activation functions on the distribution of the pre-activations and activations is investigated and discussed from both analytic and empirical viewpoints. This study, among others, shows that the pre-activation density in the bottleneck layer can be well fitted with a diagonal GMM with a few Gaussians and how and why the ReLU activation function promotes sparsity. Motivated by the statistical properties of the pre-activations, the usefulness of statistical normalisation of bottleneck features was also investigated. To this end, methods such as mean(-variance) normalisation, Gaussianisation, and histogram equalisation (HEQ) were employed and up to 2% (absolute) WER reduction achieved in the Aurora-4 task.