EAGER: Identifying and Producing Code-Switching in Languages from Spoken, Lexical and Socio-linguistic Features

这项早期的探索性研究拨款调查了世界上大量会说多种语言的人之间的对话，这些人经常在这些语言之间来回切换，这就是所谓的代码转换。重要的是，语音对话系统和语音助理不仅能够识别代码转换发生的时间、原因和效果，而且能够正确地解释所说的话，并且能够在与这样的用户交互时产生类似的代码转换响应。近年来语音技术的进步导致了语音助手的广泛使用，如Siri、谷歌助手和Alexa。它们极大地改善了英语、法语、德语、广东话、普通话和西班牙语等语言的语音信息访问。然而，这种访问仅限于单语语音，对于许多多语种演讲者来说，这不是最自然的语音产生形式。因此，语码转换的语音很少被正确理解，并且永远不能在辅助回答中产生。使这些说话者能够进行自然和舒适的交流的一个主要障碍是缺乏语音技术，这种技术不仅可以理解代码转换的输入，而且可以产生类似人类的输出。这个项目通过研究口语和书面语的代码转换如何与语言交流的其他方面相互作用来解决这些问题。它将探索语码转换研究中尚未研究的问题，包括：(1)说话者是否在语音和其他语码转换策略上牵连、说话更相似；(2)语码转换和言语中的移情之间是否存在可量化的关系，其中移情是说话者传达他们理解他人的问题并希望帮助解决这些问题的意图；(3)命名实体的存在，如姓名或地理位置，是否启动了语码转换；(4)在语码转换的言语中，哪些对话行为，如疑问句或反向通道，往往最常产生；以及(5)当说话者进行代码转换时，他们是如何产生语调轮廓的(通过选择他们的语调产物来匹配他们所说的两种语言中的一种或两种都不同？)统计和机器学习技术将在标准美国英语、西班牙语、普通话和印地语的口语和词汇特征标记的代码转换语音的背景下被用来解决这些问题。通过识别代码转换的新方面，该项目将推动研究界对这一现象的进一步探索。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This Early Grant for Exploratory Research investigates conversations between the vast number of persons in our world who speak multiple languages and who frequently switch back and forth between those languages in what is called “code-switching”. It is important for speech dialogue systems and voice assistants to not only be able to identify when, why, and to what effect code-switching occurs, but also to correctly interpret what is said and to be able to generate similarly code-switched responses when interacting with such users. Advances in speech technology in recent years have resulted in widespread use of voice assistants such as Siri, Google Assistant and Alexa. They enable vast improvement in information access by voice for languages such as English, French, German, Cantonese, Mandarin, and Spanish. However, such access is limited to monolingual speech, which for many multilingual speakers is not the most natural form of speech production. Thus, code-switched speech is rarely understood correctly and is never able to be produced in assistant responses. A major barrier to enabling naturalistic and comfortable communication for these speakers is the lack of speech technology that can not only understand code-switched input but also produce similar human-like output. This project addresses these issues by examining how spoken and written code-switching interacts with other aspects of language communication. It will explore research questions not yet studied in code-switching research including (1) whether speakers entrain, speak more similarly, on pronunciation and other strategies of code-switching in speech; (2) whether there is a quantifiable relationship between code-switching and empathy in speech, where empathy is a speaker's intention to convey that they understand another's problems and want to help address them; (3) whether the presence of named entities, such as names or geographical locations, primes code-switching; (4) which dialogue acts, such as questions or statements or backchannels, tend to be produced most often in code-switched speech; and (5) how speakers produce intonational contours when they code-switch (via choosing their intonation production to match either of the languages they are producing or by being different from both?) Statistical and machine-learning techniques will both be used to address these questions in the context of spoken and lexical-feature-tagged code-switched speech in Standard American English, Spanish, Mandarin Chinese, and Hindi. By identifying new aspects of code-switching, the project will seed further exploration of this phenomenon by the research community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.