CCRI: ENS: Next Generation Tools for Spoken Language Science & Technology

自动语音识别（ASR）和口语理解任务几乎体现了人工智能（AI）的所有要素。可靠的ASR（当无处不在时）将成为人机交互口语对话系统中强大的智能研究的关键促成因素，基于内容的多媒体搜索和口述历史档案访问的信息集成研究，以及基础语音科学和技术，以实现儿童认知发展，语言学，智能健康，老年人护理，教育以及（广泛地）行为和社会动态的机器辅助研究。该项目是在与语音和语言研究社区广泛磋商后开发的，正在广泛修订Kaldi开源工具包，以(a)使语音识别初学者和其他领域的研究人员更容易使用语音识别，(b)利用现有的深度学习框架（主要是PyTorch）来增加其灵活性，(c)创建新的用户培训材料，(d)继续增强工具包。从而支持社区内部的发展与合作。该项目在通用AI/深度学习框架（主要是PyTorch）中实现了所有核心Kaldi功能（例如，无格最大互信息训练目标），因此可以无缝地利用深度学习的相关进展（例如，新颖的优化算法）。此外，该项目通过有限状态传感器集成了自动区分，这是Kaldi的核心功能，负责其最先进的性能，允许真正的端到端ASR系统训练。这些和其他增强功能将使实现两个目前不兼容的目标成为可能：将结构外部知识（例如，对话流模型，有限状态语法，发音词汇）整合到全神经ASR系统中，以及通过反向传播对混合ASR系统进行端到端训练。该提案的其他目标包括为ASR系统的大规模训练提供高效且用户友好的数据准备和模型管理工具，以及将ASR作为其他科学研究工具的研究人员所需的强大对话分析和说话人分类能力。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The task of automatic speech recognition (ASR) and spoken language understanding embodies almost all the elements of artificial intelligence (AI). Reliable ASR (when ubiquitously available) will be a key enabler of robust intelligence research in spoken dialog systems for human-computer interactions, information integration research in content-based multimedia search and access to oral history archives, and fundamental speech science and technology to enable research in children's cognitive development, linguistics, smart health, elderly care, education, and (broadly) the machine-aided study of behavioral and social dynamics. This project, developed after extensive consultations with the speech and language research community, is extensively revising the Kaldi open-source toolkit to (a) make speech recognition more accessible both for beginners in speech recognition and researchers in other fields, (b) leverage existing deep learning framework (primarily PyTorch) to increase its flexibility, (c) create new user training materials, and (d) continue to enhance the toolkit, so as to support the growth of and cooperation within the community.The project implements all core Kaldi functions (e.g., the lattice-free maximum mutual information training objective) natively in generic AI/deep learning frameworks, primarily PyTorch, so that associated advances in deep learning (e.g., novel optimization algorithms) can be seamlessly leveraged. Furthermore, the project incorporates automatic differentiation through finite state transducers, a core Kaldi feature responsible for its state-of-the-art performance, permitting true end-to-end training of ASR systems. These and other enhancements will make it possible to achieve two currently incompatible goals: incorporating structure external knowledge (e.g., dialog flow models, finite state grammars, pronunciation lexicons) into fully neural ASR systems, and end-to-end training of a hybrid ASR system via backpropagation. Other goals of this proposal include the provision of efficient yet user-friendly data preparation and model management tools for large scale training of ASR systems, and capabilities for robust conversation analysis and speaker diarization needed by researchers who use ASR as a tool for other scientific inquiries.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.