NSF Convergence Accelerator Track D: Data & AI Methods for Modeling Facial Expressions in Language with Applications to Privacy for the Deaf, ASL Education & Linguistic Res

NSF 融合加速器轨道 D：数据

• 批准号: 2040638
• 负责人: Dimitris Metaxas
• 金额: $ 96万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-15 至 2022-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2040638&HistoricalAwards=false
• 关键词: NSF; Convergence; Accelerator; Track; Data

The NSF Convergence Accelerator supports use-inspired, team-based, multidisciplinary efforts that address challenges of national importance and will produce deliverables of value to society in the near future. American Sign Language (ASL) is the third most studies “foreign” language in the United States. This project is building 4-dimensional face-tracking algorithms that could be used to separate facial geometry from facial movement and expression. The work supports an application for teaching American Sign Language (ASL) to ASL-learners, an application for anonymizing the signer when privacy is a concern, and research into the role of facial expressions in both sign and spoken language. The privacy preserving application being developed by this project will enable ASL speakers to have private conversations about sensitive topics they would otherwise. This team of linguists, computer scientists, deaf and hearing experts on ASL, and industry partners will address research and societal challenges through three types of deliverables targeted to diverse user and research communities: 1) Modifications and extension of AI methods and publicly shared ASL data and tools to encompass spoken language. Although facial expressions and head gestures, essential to the grammar of signed languages, also play an important role in speech, this is not well understood because resources of the kind developed by this project have not been available. New data and analyses will open the door to comparative study of the role of facial expressions across modalities, and the role of facial expressions in signed language vs. spoken language. Shared raw data, analyses, and visualizations will open up new avenues for linguistic and computer science research into the role of spatiotemporal synchronization of nonmanual expressions in conjunction with speech and signing. 2) An application to help ASL learners produce facial expressions and head gestures to convey grammatical information in signed languages; and 3) Development of a tool for real-time anonymization of ASL videos to preserve grammatical information expressed non-manually, while de-identifying the signer.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

NSF融合加速器支持以使用为灵感、以团队为基础的多学科努力，以应对国家重要性的挑战，并将在不久的将来产生对社会有价值的成果。美国手语(ASL)是美国学习次数第三多的“外语”。该项目正在构建4维人脸跟踪算法，可以用来将面部几何形状与面部运动和表情分开。这项工作支持一个向美国手语学习者教授美国手语(ASL)的应用程序，一个在隐私受到关注时匿名的应用程序，以及研究面部表情在手语和口语中的作用。该项目正在开发的隐私保护应用程序将使美国手语演讲者能够就敏感话题进行私人对话。这个由语言学家、计算机科学家、聋人和听力专家组成的ASL团队以及行业合作伙伴将通过针对不同用户和研究社区的三种类型的交付成果来应对研究和社会挑战：1)修改和扩展人工智能方法，并公开共享ASL数据和工具，以涵盖口语。虽然面部表情和头势对手语的语法至关重要，但在言语中也发挥着重要作用，但这一点还没有得到很好的理解，因为该项目开发的这类资源尚未获得。新的数据和分析将打开比较研究面部表情在各种模式中的作用，以及面部表情在手语和口语中的作用的大门。共享原始数据、分析和可视化将为语言学和计算机科学研究非手动表达与语音和手势相结合的时空同步的作用开辟新的途径。2)一款应用程序，帮助美国手语学习者制作面部表情和头部手势，以手语传达语法信息；以及3)开发一种工具，用于实时匿名美国手语视频，以保留非手动表达的语法信息，同时取消签名者的身份。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Isolated Sign Recognition using ASL Datasets with Consistent Text-based Gloss Labeling and Curriculum Learning

• 发表时间: 2022
• 作者: Konstantinos M. Dafnis;Evgenia Chroni;C. Neidle;Dimitris N. Metaxas

ASL Video Corpora & Sign Bank: Resources Available through the American Sign Language Linguistic Research Project (ASLLRP)

• 发表时间: 2022-01
• 期刊: ArXiv
• 作者: C. Neidle;Augustine Opoku;Dimitris N. Metaxas

American Sign Language Video Anonymization to Support Online Participation of Deaf and Hard of Hearing Users

• DOI: 10.1145/3441852.3471200
• 发表时间: 2021-10
• 期刊: Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility
• 作者: Sooyeon Lee;Abraham Glasser;Becca Dingman;Zhaoyang Xia;Dimitris N. Metaxas;C. Neidle;Matt Huenerfauth

Understanding ASL Learners’ Preferences for a Sign Language Recording and Automatic Feedback System to Support Self-Study

了解 ASL 学习者对支持自学的手语录音和自动反馈系统的偏好

• DOI: 10.1145/3517428.3550367
• 发表时间: 2022
• 期刊: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility
• 作者: Hassan, Saad;Lee, Sooyeon;Metaxas, Dimitris;Neidle, Carol;Huenerfauth, Matt
• 通讯作者: Huenerfauth, Matt

Resources for Computer-Based Sign Recognition from Video, and the Criticality of Consistency of Gloss Labeling across Multiple Large ASL Video Corpora

• 发表时间: 2022
• 作者: C. Neidle;Augustine Opoku;Carey M. Ballard;Konstantinos M. Dafnis;Evgenia Chroni;Dimitris N. Metaxas