CAREER: Experimental pragmatics and semantics in visual language

人类语言之所以如此强大的一个原因是，它有限的词汇量可以用来分享无限多的想法，以及说出以前从未说过的话。理解单词如何组合在一起以可预测的(因此是无限的)方式表达新的含义一直是几十年来研究的重点，并导致人工智能系统在回答直接询问(如问路、天气等)方面越来越成功。然而，在面对面的互动中，语言的意义受到不仅是单词，甚至超过语调的因素的影响：非语言的视觉信息与语音清晰而有规律地交互。例如，当有人说他们一边移动手指，就像在键盘上打字一样，一边“拿着笔记”，这意味着这个笔记是电子的；另一个例子是，当说“我的邻居”时，指着某人的手势识别出这个人的标签。目前有技术可以识别这些动作中的大多数，但为了表达意义，它们通常被忽略了，因为人们对它们如何构成和与句子的其他部分组成知之甚少。这个问题的一个有价值的见解来自完全视觉的语言：美国手语等手语有自己完整的独特语法和自己复杂的词义组合规则，同时清晰而有规律地将视觉演示和指向融入到句子意义中。该项目比较了视觉(演示和指向)信息在英语手势中的作用，以及美国手语中有充分记录的对应信息。它建立在项目团队目前在手语语言学和手势理论模型方面的工作基础上，扩展到包括最先进的实验方法和分析。我们将同时研究英语和美国手语，方法是通过对在线视频的回应收集定量数据，以及分析流利的语言作品。一个主要的挑战是，研究单词如何组合成更复杂的含义的组成过程需要在逻辑和计算方面的平凡背景，而拥有这一技能的学者往往与聋人、重听人和流利的手语学者有有限的重叠，这些人对手势语言学和手势这两个迅速发展的领域最有洞察力。为了弥补这一差距，该项目将包括为研究手语和手势的博士后、研究生和本科生提供培训机会，以提供实验方法论、统计分析以及逻辑和计算方面的经验。该项目还将包括开发一门致力于弥合这一差距的课程和教科书。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

One thing that makes human language so powerful is that its limited vocabulary can be used to share an unlimited number of ideas, and to say things that have never been said before. Understanding how words combine to express new meanings in a predictable (and therefore unlimited) way has been the focus of decades of research, and has led to increasing success by artificial intelligence systems in responding to direct queries, such as asking for directions, weather, etc. However, the meaning of language in face-to-face interactions is influenced by factors beyond just words, and even beyond intonation: nonlinguistic visual information interacts clearly and regularly with speech. For example, when someone says they "took a note" while moving their fingers as if they were typing on a keyboard, this conveys that the note was electronic; as another example, a pointing gesture toward someone while saying "my neighbor" identifies the person with the label. Technology currently exists to recognize most of these movements, but they are typically ignored for the purpose of meaning since so little is known about how they contribute and compose with other parts of a sentence. A source of valuable insight for this question comes from languages which are entirely visual: sign languages such as American Sign Language have their own full distinct grammar and their own complex rules for the meanings of word combinations, and at the same time clearly and regularly integrate visual demonstrations and pointing into sentence meaning. This project compares the role of visual (demonstrations and pointing) information in gestures used in English with well-documented counterparts in American Sign Language. It builds on current work by the project team in theoretical models of sign language linguistics and gesture, expanding to include state-of-the-art experimental methodologies and analysis. English and American Sign Language will be studied at each point in parallel, both by gathering quantitative data using responses to videos presented online and by analyzing fluent language productions. A major challenge is that research on the compositional processes that underlie how words combine into more complex meanings requires a dolid background in logic and computation, and scholars with this skill set tend to have limited overlap with the Deaf, Hard of Hearing, and fluent signing scholars who have the most insight into the quickly growing fields of sign linguistics and gesture. To correct for this gap, the project will include training opportunities for postdoctoral, graduate students and undergraduates whose studies are focused on sign languages and gesture, to provide experience with experimental methodology, statistical analysis, and logic and computation. The project will also include the development of a course and textbook dedicated to bridging this gap.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.