CAREER: Experimental pragmatics and semantics in visual language

人类语言如此强大的一个原因是，它有限的词汇可以用来分享无限的想法，说以前从未说过的话。理解单词如何以一种可预测的（因此是无限的）方式组合起来表达新含义一直是数十年研究的重点，并导致人工智能系统在回应直接查询（如问路、天气等）方面越来越成功。然而，在面对面的互动中，语言的意义不仅仅受到词语甚至语调的影响：非语言的视觉信息与言语清晰而有规律地相互作用。例如，当有人一边说他们“记了个笔记”一边移动手指，就好像他们在键盘上打字一样，这就传达了这个笔记是电子的；另一个例子是，在说“我的邻居”时，用手指着某人的手势来识别这个人。目前存在的技术可以识别大多数这些动作，但由于对它们如何贡献和与句子的其他部分组成知之甚少，因此它们通常被忽略。对于这个问题，一个有价值的见解来自于完全视觉化的语言：像美国手语这样的手语有自己完全不同的语法和自己复杂的单词组合意义规则，同时清晰而有规律地将视觉演示和指向融入句子意义。这个项目比较了英语手势中视觉（演示和指向）信息的作用，以及美国手语中有充分记录的对应信息。它以项目团队目前在手语语言学和手势理论模型方面的工作为基础，扩展到包括最先进的实验方法和分析。英语和美国手语将在每个点并行学习，通过对在线视频的反应收集定量数据，并通过分析流利的语言作品。一个主要的挑战是，研究单词如何组合成更复杂的意思的构成过程需要扎实的逻辑和计算背景，而拥有这一技能的学者往往与聋哑人、重听人和流利的手语学者有有限的重叠，这些学者对快速发展的手语语言学和手势领域最有洞察力。为了弥补这一差距，该项目将为研究手语和手势的博士后、研究生和本科生提供培训机会，以提供实验方法、统计分析、逻辑和计算方面的经验。该项目还将包括开发一门课程和教科书，致力于弥合这一差距。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

One thing that makes human language so powerful is that its limited vocabulary can be used to share an unlimited number of ideas, and to say things that have never been said before. Understanding how words combine to express new meanings in a predictable (and therefore unlimited) way has been the focus of decades of research, and has led to increasing success by artificial intelligence systems in responding to direct queries, such as asking for directions, weather, etc. However, the meaning of language in face-to-face interactions is influenced by factors beyond just words, and even beyond intonation: nonlinguistic visual information interacts clearly and regularly with speech. For example, when someone says they "took a note" while moving their fingers as if they were typing on a keyboard, this conveys that the note was electronic; as another example, a pointing gesture toward someone while saying "my neighbor" identifies the person with the label. Technology currently exists to recognize most of these movements, but they are typically ignored for the purpose of meaning since so little is known about how they contribute and compose with other parts of a sentence. A source of valuable insight for this question comes from languages which are entirely visual: sign languages such as American Sign Language have their own full distinct grammar and their own complex rules for the meanings of word combinations, and at the same time clearly and regularly integrate visual demonstrations and pointing into sentence meaning. This project compares the role of visual (demonstrations and pointing) information in gestures used in English with well-documented counterparts in American Sign Language. It builds on current work by the project team in theoretical models of sign language linguistics and gesture, expanding to include state-of-the-art experimental methodologies and analysis. English and American Sign Language will be studied at each point in parallel, both by gathering quantitative data using responses to videos presented online and by analyzing fluent language productions. A major challenge is that research on the compositional processes that underlie how words combine into more complex meanings requires a dolid background in logic and computation, and scholars with this skill set tend to have limited overlap with the Deaf, Hard of Hearing, and fluent signing scholars who have the most insight into the quickly growing fields of sign linguistics and gesture. To correct for this gap, the project will include training opportunities for postdoctoral, graduate students and undergraduates whose studies are focused on sign languages and gesture, to provide experience with experimental methodology, statistical analysis, and logic and computation. The project will also include the development of a course and textbook dedicated to bridging this gap.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.