Collaborative Research: Quantifying sign reduction in sign language using human pose estimation

本项目研究了手语中词汇条件下的语音变异现象。这指的是一个符号的发音如何根据符号的属性而变化，比如在心理词汇中出现的频率或发音相似的符号（称为语音邻居）的存在。在口语中，经常使用的单词在语音上可能会被弱化，也就是说，与低频单词相比，它们的元音空间更短、收缩。另一方面，有许多语音邻居的单词可能会在语音上得到增强，例如，与更孤立的单词相比，产生的元音空间更大。发声器（声道）与发声器（手、脸、身体）有着根本的不同。这种情态差异可能导致对语音变化的不同预测。关于这些原则如何应用于手语的研究有限，主要是由于缺乏大规模的机器可读手语数据集和从手语视频中提取语音测量的足够技术。在美国国家科学基金会（NSF）之前的支持下，研究人员利用计算机视觉和手语词汇数据库来分析超过10万段聋哑人制作的手语视频中的语音减少模式。具体来说，他们使用人体姿势估计技术从视频中提取出签名者的解剖标志和关节位置，并将这些估计转化为有意义的语音测量，例如签名者的手在签名空间中的分散。通过研究符号的内在和表面特性之间的相互作用，本研究为手语的语言结构如何影响符号发音提供了见解。本研究通过对手语语音特性的了解，促进手语识别技术的发展，造福于手语界。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The project investigates the phenomenon of lexically conditioned phonetic variation in sign language. This refers to how the pronunciation of a sign can vary based on the properties of signs, such as frequency of occurrence or the existence of similar-sounding signs (referred to as phonological neighbors) in the mental lexicon. In spoken languages, words that are used frequently may be phonetically reduced, that is, articulated with shorter, contracted vowel space as compared to low-frequency words. On the other hand, words with many phonological neighbors may be phonetically enhanced, e.g., produced with expanded vowel space compared to more isolated words. The articulators used to produce speech (the vocal tract) are fundamentally different from those used to produce signs (hands, face, body). This modality difference may lead to distinct predictions about phonetic variation. Limited research has been done on how these principles apply in signed languages, primarily due to a lack of large-scale machine-readable sign language datasets and adequate techniques to extract phonetic measurements from signed videos. The researchers leverage computer vision and the lexical database for a sign language that they developed under prior National Science Foundation (NSF) support, to analyze patterns of phonetic reduction in over 100,000 videos of signs produced by deaf signers. Specifically, they use human pose estimation techniques to extract signers’ anatomical landmarks and joint positions from the videos and transform the estimates into meaningful phonetic measurements, such as dispersion of the signers’ hands in the signing space. By examining the interactions between the underlying and surface properties of signs, this research provides insights into how the linguistic structure of sign languages affects sign articulation. By understanding the phonetic properties of signs, this research facilitates the development of sign recognition technologies, benefiting the signing community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.