EAGER: Identifying and Producing Code-Switching in Languages from Spoken, Lexical and Socio-linguistic Features

这项探索性研究的早期拨款调查了我们这个世界上讲多种语言的大量人之间的对话，这些人经常在这些语言之间来回切换，这被称为“代码转换”。语音对话系统和语音助手不仅要能够识别代码转换发生的时间、原因和效果，而且要正确解释所说的内容，并能够在与此类用户交互时生成类似的代码转换响应，这一点很重要。近年来，语音技术的进步导致了Siri、b谷歌Assistant和Alexa等语音助手的广泛使用。它们极大地改善了英语、法语、德语、广东话、普通话和西班牙语等语言的语音信息获取。然而，这种访问仅限于单语语音，对于许多多语使用者来说，单语语音并不是最自然的语音产生形式。因此，代码转换语音很少被正确理解，并且永远无法在辅助响应中产生。对于这些说话者来说，实现自然和舒适的交流的一个主要障碍是缺乏语音技术，这种技术不仅可以理解代码切换的输入，还可以产生类似人类的输出。本项目通过考察口语和书面语代码转换如何与语言交流的其他方面相互作用来解决这些问题。它将探讨语码转换研究中尚未研究的研究问题，包括(1)说话者是否在语音中的发音和其他语码转换策略上更相似；(2)语码转换与言语共情之间是否存在可量化的关系，其中共情是说话者意图传达他们理解他人的问题并希望帮助解决问题；(3)名称实体（如名称或地理位置）的存在是否会促进代码转换；(4)在语码转换言语中，哪些对话行为（如提问、陈述或反向通道）最常发生；(5)说话者在进行语码转换时是如何产生语调轮廓的（通过选择语调产生来匹配他们所产生的其中一种语言，还是通过与两种语言都不同？）统计和机器学习技术都将用于在标准美式英语、西班牙语、普通话和印地语的口语和词汇特征标记的代码转换语音环境中解决这些问题。通过识别代码转换的新方面，该项目将为研究界对这一现象的进一步探索奠定基础。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This Early Grant for Exploratory Research investigates conversations between the vast number of persons in our world who speak multiple languages and who frequently switch back and forth between those languages in what is called “code-switching”. It is important for speech dialogue systems and voice assistants to not only be able to identify when, why, and to what effect code-switching occurs, but also to correctly interpret what is said and to be able to generate similarly code-switched responses when interacting with such users. Advances in speech technology in recent years have resulted in widespread use of voice assistants such as Siri, Google Assistant and Alexa. They enable vast improvement in information access by voice for languages such as English, French, German, Cantonese, Mandarin, and Spanish. However, such access is limited to monolingual speech, which for many multilingual speakers is not the most natural form of speech production. Thus, code-switched speech is rarely understood correctly and is never able to be produced in assistant responses. A major barrier to enabling naturalistic and comfortable communication for these speakers is the lack of speech technology that can not only understand code-switched input but also produce similar human-like output. This project addresses these issues by examining how spoken and written code-switching interacts with other aspects of language communication. It will explore research questions not yet studied in code-switching research including (1) whether speakers entrain, speak more similarly, on pronunciation and other strategies of code-switching in speech; (2) whether there is a quantifiable relationship between code-switching and empathy in speech, where empathy is a speaker's intention to convey that they understand another's problems and want to help address them; (3) whether the presence of named entities, such as names or geographical locations, primes code-switching; (4) which dialogue acts, such as questions or statements or backchannels, tend to be produced most often in code-switched speech; and (5) how speakers produce intonational contours when they code-switch (via choosing their intonation production to match either of the languages they are producing or by being different from both?) Statistical and machine-learning techniques will both be used to address these questions in the context of spoken and lexical-feature-tagged code-switched speech in Standard American English, Spanish, Mandarin Chinese, and Hindi. By identifying new aspects of code-switching, the project will seed further exploration of this phenomenon by the research community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.