EAGER: Identifying and Producing Code-Switching in Languages from Spoken, Lexical and Socio-linguistic Features

这项早期探索性研究资助调查了世界上大量讲多种语言的人之间的对话，这些人经常在这些语言之间来回切换，即所谓的“语码转换”。对于语音对话系统和语音助手来说，重要的是不仅能够识别语码转换发生的时间、原因和效果，而且能够正确解释所说的内容，并在与此类用户交互时能够生成类似的语码转换响应。近年来语音技术的进步导致了 Siri、Google Assistant 和 Alexa 等语音助手的广泛使用。它们极大地改善了英语、法语、德语、粤语、普通话和西班牙语等语言的语音信息访问。然而，这种访问仅限于单语语音，对于许多多语言使用者来说，这并不是最自然的语音产生形式。因此，语码转换语音很少能被正确理解，并且永远无法在助理响应中产生。让这些说话者实现自然、舒适的沟通的一个主要障碍是缺乏不仅能够理解代码转换输入而且能够产生类似人类输出的语音技术。该项目通过研究口语和书面语码转换如何与语言交流的其他方面相互作用来解决这些问题。它将探讨语码转换研究中尚未研究的研究问题，包括（1）说话者是否在语音和其他语音语码转换策略上夹带、说话更相似； （2）言语中的语码转换和同理心之间是否存在可量化的关系，其中同理心是说话者表达自己理解他人问题并希望帮助解决问题的意图； (3) 命名实体（例如名称或地理位置）的存在是否会引发语码转换； (4) 哪些对话行为，例如问题、陈述或秘密渠道，最常在语码转换语音中产生； （5）说话者在语码转换时如何产生语调轮廓（通过选择语调产生来匹配他们正在产生的任何一种语言，还是与两种语言都不同？）统计和机器学习技术都将用于在标准美式英语、西班牙语、普通话和印地语的口语和词汇特征标记语码转换语音的背景下解决这些问题。通过确定语码转换的新方面，该项目将推动研究界对这一现象的进一步探索。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

This Early Grant for Exploratory Research investigates conversations between the vast number of persons in our world who speak multiple languages and who frequently switch back and forth between those languages in what is called “code-switching”. It is important for speech dialogue systems and voice assistants to not only be able to identify when, why, and to what effect code-switching occurs, but also to correctly interpret what is said and to be able to generate similarly code-switched responses when interacting with such users. Advances in speech technology in recent years have resulted in widespread use of voice assistants such as Siri, Google Assistant and Alexa. They enable vast improvement in information access by voice for languages such as English, French, German, Cantonese, Mandarin, and Spanish. However, such access is limited to monolingual speech, which for many multilingual speakers is not the most natural form of speech production. Thus, code-switched speech is rarely understood correctly and is never able to be produced in assistant responses. A major barrier to enabling naturalistic and comfortable communication for these speakers is the lack of speech technology that can not only understand code-switched input but also produce similar human-like output. This project addresses these issues by examining how spoken and written code-switching interacts with other aspects of language communication. It will explore research questions not yet studied in code-switching research including (1) whether speakers entrain, speak more similarly, on pronunciation and other strategies of code-switching in speech; (2) whether there is a quantifiable relationship between code-switching and empathy in speech, where empathy is a speaker's intention to convey that they understand another's problems and want to help address them; (3) whether the presence of named entities, such as names or geographical locations, primes code-switching; (4) which dialogue acts, such as questions or statements or backchannels, tend to be produced most often in code-switched speech; and (5) how speakers produce intonational contours when they code-switch (via choosing their intonation production to match either of the languages they are producing or by being different from both?) Statistical and machine-learning techniques will both be used to address these questions in the context of spoken and lexical-feature-tagged code-switched speech in Standard American English, Spanish, Mandarin Chinese, and Hindi. By identifying new aspects of code-switching, the project will seed further exploration of this phenomenon by the research community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.