Collaborative Research: Quantifying sign reduction in sign language using human pose estimation

合作研究：使用人体姿势估计量化手语中的符号减少

• 批准号: 2234787
• 负责人: Naomi Caselli
• 金额: $ 20.81万
• 依托单位: Trustees of Boston University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2234787&HistoricalAwards=false
• 关键词: Collaborative; Research; Quantifying; sign; reduction

The project investigates the phenomenon of lexically conditioned phonetic variation in sign language. This refers to how the pronunciation of a sign can vary based on the properties of signs, such as frequency of occurrence or the existence of similar-sounding signs (referred to as phonological neighbors) in the mental lexicon. In spoken languages, words that are used frequently may be phonetically reduced, that is, articulated with shorter, contracted vowel space as compared to low-frequency words. On the other hand, words with many phonological neighbors may be phonetically enhanced, e.g., produced with expanded vowel space compared to more isolated words. The articulators used to produce speech (the vocal tract) are fundamentally different from those used to produce signs (hands, face, body). This modality difference may lead to distinct predictions about phonetic variation. Limited research has been done on how these principles apply in signed languages, primarily due to a lack of large-scale machine-readable sign language datasets and adequate techniques to extract phonetic measurements from signed videos.The researchers leverage computer vision and the lexical database for a sign language that they developed under prior National Science Foundation (NSF) support, to analyze patterns of phonetic reduction in over 100,000 videos of signs produced by deaf signers. Specifically, they use human pose estimation techniques to extract signers’ anatomical landmarks and joint positions from the videos and transform the estimates into meaningful phonetic measurements, such as dispersion of the signers’ hands in the signing space. By examining the interactions between the underlying and surface properties of signs, this research provides insights into how the linguistic structure of sign languages affects sign articulation. By understanding the phonetic properties of signs, this research facilitates the development of sign recognition technologies, benefiting the signing community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

本研究旨在探讨手语中受词汇制约的语音变异现象。这是指符号的发音如何根据符号的属性而变化，例如出现的频率或心理词典中相似发音符号（称为语音邻居）的存在。在口语中，频繁使用的单词可能在语音上被减少，即，与低频单词相比，用较短的、收缩的元音空间来发音。另一方面，具有许多音位相邻词的词可以在语音上被增强，例如，与更孤立的单词相比，元音空间扩大。用于产生语音的发音器官（声道）与用于产生手势的发音器官（手、脸、身体）有着根本的不同。这种模态差异可能导致语音变异的不同预测。关于这些原则如何应用于手语的研究有限，主要是由于缺乏大规模的机器可读手语数据集和足够的技术来从签名视频中提取语音测量。研究人员利用他们在美国国家科学基金会（NSF）支持下开发的计算机视觉和手语词汇数据库，分析了100多个手语中的语音减少模式，000个聋人手语视频。具体来说，他们使用人体姿势估计技术从视频中提取签名者的解剖标志和关节位置，并将估计值转换为有意义的语音测量值，例如签名者的手在签名空间中的分散度。通过研究符号的底层和表层属性之间的相互作用，本研究提供了深入了解手语的语言结构如何影响符号清晰度。通过理解符号的语音特性，这项研究促进了符号识别技术的发展，使签名社区受益。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。