RI:Small: Nonlinear signal representations for speech applications

人类的语言是一个非常丰富的信号。除了单词，它还包含有关说话人的几种重要信息，如身份、性别、年龄、母语、方言和情感。它还提供有关传输通道和环境的信息；例如，语音是来自电话还是高保真录音，以及是否存在背景噪音。该项目旨在创建一种强大的统一表示法，以反映语音携带的所有信息。这样的表示将使重要的语音应用在社会的多个部门：商业(安全、医疗保健、用户界面)、政府(安全、信息过滤)和执法(说话人识别、取证)。在这个项目中，约翰·霍普金斯大学的研究人员发明了最初的I-向量框架，他们打算通过研究非线性模型来超越线性I-向量方法，期望更好地解释语音的复杂结构。为了实现这一目标，研究了两种不同的模型。首先，探讨了一种非线性I向量形式。在该方法中，语音信号的分布用高斯混合模型(GMM)建模。由GMM均值形成的超矢量是潜在变量(语音表示)的非线性函数(神经网络)。神经网络的参数和潜在表示的参数可以通过随机梯度下降迭代来联合估计。其次，该团队打算调查不同类型的自动编码器网络(AE、VAE、RBM、DBM)，以从它们的隐藏层获得表示。初步研究表明，通过组合来自多个隐含层的激活来获得良好的表示是可行的。可视化工具用于了解语音数据是如何表示和组织的。通过了解通过自动编码器网络建模和使用可视化工具创建的非线性关系，有可能对语音建模产生有价值的见解。这些见解可以帮助认知科学和神经科学的研究人员理解大脑是如何代表语音信号的。建议的方法是作为软件开发的，该软件将语音片段作为输入，并生成单个向量，该向量可用于描述上一段中提到的重要应用的片段特征。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Human speech is a very rich signal. In addition to words, it contains several kinds of important information about the speaker such as identity, gender, age, native language, dialect, and emotion. It also provides information about the transmission channel and environment; for example, whether the speech came from a phone call or a high-fidelity recording, and whether or not there was background noise. This project aims to create a powerful uniform representation that reflects all the information carried by speech. Such representation would enable important speech applications in multiple sectors of society: commercial (security, healthcare, user interfaces), government (security, information filtering), and law enforcement (speaker identification, forensics).In this project, Johns Hopkins University researchers, who invented the original i-vector framework, intend to progress beyond the linear i-vector approach by investigating non-linear models with the expectation to better explain the complex structure of speech. To achieve this goal, two different models are investigated. First, a non-linear i-vector version is explored. In this method, the speech signal distribution is modeled by a Gaussian mixture model (GMM). The super-vector formed by the GMM means is a non-linear function (neural network) of a latent variable (speech representation). The parameters of the neural network and the latent representation can be jointly estimated by stochastic gradient descent iterations. Secondly, the team intends to investigate different types of auto-encoder networks (AE, VAE, RBM, DBM) to obtain representations from their hidden layers. Preliminary research shows that it is feasible to obtain good representations by combining activations from several hidden layers. Visualization tools are used to understand how the speech data have been represented and structured. By understanding the non-linear relationships created via the auto-encoder network modeling and using the visualization tools, there is potential to produce valuable insights into speech modeling. These insights can help cognitive science and neuroscience researchers to understand how the brain represents speech signals. The proposed methods are developed as software that takes a speech segment as input and generates a single vector that may be used to characterize the segment for the important applications mentioned in the previous paragraph.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.