CCRI: ENS: Next Generation Tools for Spoken Language Science & Technology

CCRI：ENS：下一代口语科学工具

• 批准号: 2120435
• 负责人: Sanjeev Khudanpur
• 金额: $ 184万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2120435&HistoricalAwards=false
• 关键词: CCRI; ENS; Next; Generation; Tools

The task of automatic speech recognition (ASR) and spoken language understanding embodies almost all the elements of artificial intelligence (AI). Reliable ASR (when ubiquitously available) will be a key enabler of robust intelligence research in spoken dialog systems for human-computer interactions, information integration research in content-based multimedia search and access to oral history archives, and fundamental speech science and technology to enable research in children's cognitive development, linguistics, smart health, elderly care, education, and (broadly) the machine-aided study of behavioral and social dynamics. This project, developed after extensive consultations with the speech and language research community, is extensively revising the Kaldi open-source toolkit to (a) make speech recognition more accessible both for beginners in speech recognition and researchers in other fields, (b) leverage existing deep learning framework (primarily PyTorch) to increase its flexibility, (c) create new user training materials, and (d) continue to enhance the toolkit, so as to support the growth of and cooperation within the community.The project implements all core Kaldi functions (e.g., the lattice-free maximum mutual information training objective) natively in generic AI/deep learning frameworks, primarily PyTorch, so that associated advances in deep learning (e.g., novel optimization algorithms) can be seamlessly leveraged. Furthermore, the project incorporates automatic differentiation through finite state transducers, a core Kaldi feature responsible for its state-of-the-art performance, permitting true end-to-end training of ASR systems. These and other enhancements will make it possible to achieve two currently incompatible goals: incorporating structure external knowledge (e.g., dialog flow models, finite state grammars, pronunciation lexicons) into fully neural ASR systems, and end-to-end training of a hybrid ASR system via backpropagation. Other goals of this proposal include the provision of efficient yet user-friendly data preparation and model management tools for large scale training of ASR systems, and capabilities for robust conversation analysis and speaker diarization needed by researchers who use ASR as a tool for other scientific inquiries.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

自动语音识别（ASR）和口语理解的任务几乎包含了人工智能（AI）的所有元素。 可靠的ASR（当无处不在时）将成为人机交互口语对话系统中强大智能研究、基于内容的多媒体搜索和口述历史档案访问中的信息集成研究以及基础语音科学和技术的关键推动者，以实现儿童认知发展、语言学、智能健康、老年人护理、教育、以及（广义上）对行为和社会动力学的机器辅助研究。该项目是在与语音和语言研究社区进行广泛磋商后开发的，它正在广泛修订Kaldi开源工具包，以（a）使语音识别初学者和其他领域的研究人员更容易使用语音识别，（B）利用现有的深度学习框架（主要是PyTorch）以增加其灵活性，（c）创建新的用户培训材料，（d）继续增强工具包，以支持社区的发展和社区内的合作。该项目实现了Kaldi的所有核心功能（例如，无格最大互信息训练目标）原生地在通用AI/深度学习框架中，主要是PyTorch，使得深度学习的相关进步（例如，新颖的优化算法）可以无缝地被利用。此外，该项目通过有限状态传感器集成了自动微分，这是Kaldi的核心功能，负责其最先进的性能，允许ASR系统的真正端到端培训。这些和其他增强将有可能实现两个目前不兼容的目标：整合结构外部知识（例如，对话流模型、有限状态语法、发音词典）到全神经ASR系统中，以及通过反向传播对混合ASR系统进行端到端训练。该提案的其他目标包括为ASR系统的大规模培训提供高效且用户友好的数据准备和模型管理工具，和强大的会话分析和发言者日记的能力，研究人员谁使用ASR作为其他科学调查的工具所需要的。这个奖项反映了NSF的法定使命，并已被认为是值得通过评估使用基金会的智力价值和更广泛的影响审查标准。

项目成果

Building Keyword Search System from End-To-End Asr Systems

• DOI: 10.1109/icassp49357.2023.10097249
• 发表时间: 2023-06
• 期刊: ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
• 作者: Ruizhe Huang;Matthew Wiesner;Leibny Paola García-Perera;Daniel Povey;J. Trmal;S. Khudanpur

Defense against Adversarial Attacks on Hybrid Speech Recognition System using Adversarial Fine-tuning with Denoiser

使用降噪器进行对抗性微调来防御混合语音识别系统的对抗性攻击

• DOI: 10.21437/interspeech.2022-10977
• 发表时间: 2022
• 期刊: Proc. Interspeech 2022
• 作者: Joshi, Sonal;Kataria, Saurabh;Shao, Yiwen;Żelasko, Piotr;Villalba, Jesús;Khudanpur, Sanjeev;Dehak, Najim
• 通讯作者: Dehak, Najim

Chunking Defense for Adversarial Attacks on ASR

• DOI: 10.21437/interspeech.2022-11096
• 发表时间: 2022-09
• 作者: Yiwen Shao;J. Villalba;Sonal Joshi;Saurabh Kataria;S. Khudanpur;N. Dehak

SURT 2.0: Advances in Transducer-Based Multi-Talker Speech Recognition

• DOI: 10.1109/taslp.2023.3318398
• 发表时间: 2023-06
• 期刊: IEEE/ACM Transactions on Audio, Speech, and Language Processing
• 作者: Desh Raj;Daniel Povey;S. Khudanpur

GPU-accelerated Guided Source Separation for Meeting Transcription

用于会议转录的 GPU 加速引导源分离

• DOI: 10.21437/interspeech.2023-42
• 发表时间: 2023
• 期刊: Proc. INTERSPEECH 2023
• 作者: Raj, Desh;Povey, Daniel;Khudanpur, Sanjeev
• 通讯作者: Khudanpur, Sanjeev