RI: Small: From acoustics to semantics: Embedding speech for a hierarchy of tasks

RI：小：从声学到语义：为任务层次结构嵌入语音

• 批准号: 1816627
• 负责人: Karen Livescu
• 金额: $ 45万
• 依托单位: Toyota Technological Institute at Chicago
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-15 至 2021-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1816627&HistoricalAwards=false
• 关键词: RI; Small; acoustics; semantics; Embedding

There is an increasingly large array of spoken language interfaces available, such as virtual assistants and telephone customer service interfaces. These technologies both (1) recognize the words spoken by a user and (2) extract actionable information, such as the topic of the user's query and the degree of match between the query and documents in a database. Such applications are typically treated as a pipeline of automatic speech transcription followed by text processing to extract the meaning. This project aims to develop technology that directly extracts meaning from speech, while using a variety of linguistic information along the way. This approach is intended to mitigate the effects of speech recognition errors, as well as to use all of the meaning-bearing information in speech, such as intonation. This work is expected to have long-term broad impact through technological advances, as well as immediate broad impact through the PI's involvement in local schools and mentoring for a diverse set of visiting students.The technical goals of this work are (1) to do high-quality natural language processing directly on speech; (2) to seamlessly integrate domain knowledge into end-to-end speech models; (3) improve the performance-vs.-resources tradeoff; and (4) develop models for embedding arbitrary speech signals into meaning-bearing representations. The process of mapping from speech to meaning can be viewed as a hierarchy of tasks, from the most basic acoustic-phonetic tasks to the deepest semantic tasks. The experimental work will focus on two task hierarchies: a "retrieval" hierarchy including query-by-example search, keyword spotting, semantic speech search; and a "recognition" hierarchy including phonetic recognition, word recognition, parsing, and topic identification. The main technical approaches to be developed include hierarchical multitask learning methods for incorporating domain knowledge and mitigating low-data settings, as well as new models for acoustic-semantic speech embedding.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

有越来越多的可用的口语接口，例如虚拟助理和电话客户服务接口。这些技术既(1)识别用户说出的单词，(2)提取可操作的信息，例如用户查询的主题以及查询与数据库中的文档之间的匹配程度。这类应用程序通常被视为自动语音转录的管道，随后进行文本处理以提取含义。该项目旨在开发直接从语音中提取含义的技术，同时在此过程中使用各种语言信息。这种方法的目的是减轻语音识别错误的影响，以及使用语音中的所有承载意义的信息，例如语调。这项工作的技术目标是(1)直接对语音进行高质量的自然语言处理；(2)将领域知识无缝地集成到端到端的语音模型中；(3)改进性能与资源的权衡；(4)开发将任意语音信号嵌入到具有意义的表示中的模型。从语音到意义的映射过程可以看作是一个任务的层次结构，从最基本的声学-语音任务到最深层的语义任务。实验工作将集中在两个任务层次上：一个是包括逐例查询、关键字识别、语义语音搜索的“检索”层次，另一个是包括语音识别、单词识别、句法分析和主题识别的“识别”层次。将开发的主要技术方法包括用于整合领域知识和缓解低数据环境的分层多任务学习方法，以及用于声学-语义语音嵌入的新模型。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Semantic Query-by-example Speech Search Using Visual Grounding

• DOI: 10.1109/icassp.2019.8683275
• 发表时间: 2019-04
• 期刊: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
• 作者: H. Kamper;Aristotelis Anastassiou;Karen Livescu

A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddings

• 发表时间: 2020-12
• 期刊: ArXiv
• 作者: Puyuan Peng;H. Kamper;Karen Livescu

Multilingual Jointly Trained Acoustic and Written Word Embeddings

• DOI: 10.21437/interspeech.2020-2828
• 发表时间: 2020-06
• 作者: Yushi Hu;Shane Settle;Karen Livescu

On the contributions of visual and textual supervision in low-resource semantic speech retrieval

视觉和文本监督在低资源语义语音检索中的贡献

• 发表时间: 2019
• 期刊: Interspeech 2019
• 作者: Pasad, A.;Shi, B.;Kamper, H.;Livescu, K.
• 通讯作者: Livescu, K.

Whole-Word Segmental Speech Recognition with Acoustic Word Embeddings

• DOI: 10.1109/slt48900.2021.9383578
• 发表时间: 2020-07
• 期刊: 2021 IEEE Spoken Language Technology Workshop (SLT)
• 作者: Bowen Shi;Shane Settle;Karen Livescu