Enhancing research on speech and deep learning through holistic acoustic analysis

通过整体声学分析加强语音和深度学习研究

• 批准号: 2219843
• 负责人: Matthew Goldrick
• 金额: $ 100万
• 依托单位: Northwestern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-08-15 至 2026-07-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2219843&HistoricalAwards=false
• 关键词: Enhancing; research; speech; deep; learning

You can guess a lot about a person from the way they pronounce words. Remarkably, human listeners can tell if it is likely that talkers learned English as a first language or a second language, or if the talkers might have a brain injury that makes it difficult for them to speak. Such intuitions rely on human listeners’ holistic pattern recognition abilities; these allow us to perceive the important, meaningful, yet subtle differences between pronunciations. However, the methods scientists currently use to measure speech objectively – based on a small number of properties of speech sounds – fail to capture these differences, hampering our ability to use speech to learn about the mind and brain. This project brings together speech scientists, computer scientists, and neuroscientists to test a radically different approach to this problem. Machine learning will be used to discover a new method for quantifying differences between spoken utterances based on holistic pattern recognition. This will be tested against new and existing data from bilingual speakers. If successful, this will yield a fully general method that can be applied to speech from any language or any domain of language usage, allowing scientists to capitalize on the wealth of information in speech to develop powerful new insights into the mind and brain. Improved detection of subtle problems with pronunciation, such as occurs with Alzheimer’s disease, will advance our understanding of the brain mechanisms that humans use to produce speech. The results of this testing will also allow computer scientists to advance our understanding of how machine learning algorithms process sounds, driving improvements in the algorithms and supporting applications in any area of speech and language technology that relies on spoken language processing. Speech variability across talkers provides a treasure trove of information for cognitive neuroscientists, leading to important insights into the cognitive mechanisms underlying language processing and potentially providing early signs of brain dysfunction. Current studies of speech are hamstrung by analyses that require preselecting specific temporal scales and acoustic dimensions. We propose a radically different approach: using unsupervised deep learning to discover a representational space for analysis of acoustic variation. To test this highly general approach, this method will be compared to current state-of-the art methods for analyzing individual variation in bilingual speech. This includes using the acoustic variation in second language speech to predict intelligibility and to detect difficulties in code-switching, particularly the challenges faced by individuals with Alzheimer’s Disease. The results will inform development of deep learning and cognitive neuroscience. The machine learning algorithm is fully general; it can be applied to speech from any language or any domain of language usage, expanding the range of populations and contexts that can be served by speech technology or studied by cognitive neuroscientists. The project’s integrative approach will allow computer scientists to advance our understanding of the extent to which modern deep learning architectures do or do not approximate human speech processing and allow cognitive neuroscientists to further our understanding of how meaningful acoustic distinctions are represented in speech perception and production. human speech representation. This project is funded by the Integrative Strategies for Understanding Neural and Cognitive Systems (NCS) program, which is jointly supported by the Directorates for Computer and Information Science and Engineering (CISE), Education and Human Resources (EHR), Engineering (ENG), and Social, Behavioral, and Economic Sciences (SBE).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

你可以从一个人发音的方式猜出很多关于他的信息。值得注意的是，人类听众可以判断说话者是否有可能将英语作为第一语言或第二语言，或者说话者是否可能患有脑损伤而导致说话困难。这种直觉依赖于人类听众的整体模式识别能力；这些使我们能够感知发音之间重要的、有意义的但微妙的差异。然而，科学家目前用来客观测量语音的方法（基于语音的少量属性）无法捕获这些差异，从而阻碍了我们使用语音来了解思想和大脑的能力。该项目汇集了语音科学家、计算机科学家和神经科学家，以测试解决此问题的完全不同的方法。机器学习将用于发现一种基于整体模式识别来量化口语之间差异的新方法。这将针对双语使用者的新数据和现有数据进行测试。如果成功，这将产生一种完全通用的方法，可以应用于任何语言或任何语言使用领域的语音，使科学家能够利用语音中的丰富信息来开发对思想和大脑的强大新见解。改进对发音细微问题（例如阿尔茨海默病）的检测，将增进我们对人类用于产生语音的大脑机制的理解。此次测试的结果还将帮助计算机科学家加深对机器学习算法如何处理声音的理解，推动算法的改进，并支持依赖口语处理的任何语音和语言技术领域的应用。说话者之间的言语变异为认知神经科学家提供了宝贵的信息宝库，使他们能够深入了解语言处理背后的认知机制，并可能提供大脑功能障碍的早期迹象。当前的语音研究因需要预先选择特定时间尺度和声学维度的分析而受到阻碍。我们提出了一种完全不同的方法：使用无监督深度学习来发现用于分析声学变化的表征空间。为了测试这种高度通用的方法，该方法将与当前用于分析双语语音个体差异的最先进方法进行比较。这包括利用第二语言语音的声学变化来预测清晰度并检测语码转换的困难，特别是阿尔茨海默病患者面临的挑战。研究结果将为深度学习和认知神经科学的发展提供信息。机器学习算法完全通用；它可以应用于任何语言或任何语言使用领域的语音，扩大语音技术可以服务或认知神经科学家研究的人群和环境的范围。该项目的综合方法将使计算机科学家能够加深我们对现代深度学习架构在多大程度上接近人类语音处理的理解，并让认知神经科学家进一步理解有意义的声学区别在语音感知和产生中是如何表示的。人类语音表示。该项目由理解神经和认知系统综合策略 (NCS) 计划资助，该计划由计算机和信息科学与工程 (CISE)、教育和人力资源 (EHR)、工程 (ENG) 以及社会、行为和经济科学 (SBE) 理事会联合支持。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力和知识进行评估，认为值得支持。 更广泛的影响审查标准。

项目成果

Inhibitory control of the dominant language: Reversed language dominance is the tip of the iceberg

• DOI: 10.1016/j.jml.2023.104410
• 发表时间: 2023-01-23
• 期刊: JOURNAL OF MEMORY AND LANGUAGE
• 影响因子: 4.3
• 作者: Goldrick, Matthew;Gollan, Tamar H.
• 通讯作者: Gollan, Tamar H.

Advancement of phonetics in the 21st century: Exemplar models of speech production

21 世纪语音学的进步：语音产生的范例模型

• 发表时间: 2023
• 期刊: Journal of Phonetics
• 影响因子: 1.9
• 作者: Goldrick, Matthew;Cole, Jennifer
• 通讯作者: Cole, Jennifer