NSF Convergence Accelerator Track D: Data & AI Methods for Modeling Facial Expressions in Language with Applications to Privacy for the Deaf, ASL Education & Linguistic Res

NSF 融合加速器轨道 D：数据

• 批准号: 2040638
• 负责人: Dimitris Metaxas
• 金额: $ 96万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-15 至 2022-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2040638&HistoricalAwards=false
• 关键词: NSF; Convergence; Accelerator; Track; Data

The NSF Convergence Accelerator supports use-inspired, team-based, multidisciplinary efforts that address challenges of national importance and will produce deliverables of value to society in the near future. American Sign Language (ASL) is the third most studies “foreign” language in the United States. This project is building 4-dimensional face-tracking algorithms that could be used to separate facial geometry from facial movement and expression. The work supports an application for teaching American Sign Language (ASL) to ASL-learners, an application for anonymizing the signer when privacy is a concern, and research into the role of facial expressions in both sign and spoken language. The privacy preserving application being developed by this project will enable ASL speakers to have private conversations about sensitive topics they would otherwise. This team of linguists, computer scientists, deaf and hearing experts on ASL, and industry partners will address research and societal challenges through three types of deliverables targeted to diverse user and research communities: 1) Modifications and extension of AI methods and publicly shared ASL data and tools to encompass spoken language. Although facial expressions and head gestures, essential to the grammar of signed languages, also play an important role in speech, this is not well understood because resources of the kind developed by this project have not been available. New data and analyses will open the door to comparative study of the role of facial expressions across modalities, and the role of facial expressions in signed language vs. spoken language. Shared raw data, analyses, and visualizations will open up new avenues for linguistic and computer science research into the role of spatiotemporal synchronization of nonmanual expressions in conjunction with speech and signing. 2) An application to help ASL learners produce facial expressions and head gestures to convey grammatical information in signed languages; and 3) Development of a tool for real-time anonymization of ASL videos to preserve grammatical information expressed non-manually, while de-identifying the signer.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

NSF融合加速器支持以使用为灵感，以团队为基础，多学科的努力，以应对国家重要性的挑战，并将在不久的将来为社会提供有价值的成果。美国手语（ASL）是美国第三大学习“外国”语言。该项目正在构建4维面部跟踪算法，可用于将面部几何形状与面部运动和表情分开。这项工作支持美国手语（ASL）教学的应用程序，美国手语学习者，匿名签名时，隐私是一个问题的应用程序，并研究面部表情在手语和口语中的作用。该项目正在开发的隐私保护应用程序将使美国手语发言者能够就敏感话题进行私人对话。该团队由语言学家、计算机科学家、ASL聋人和听力专家以及行业合作伙伴组成，将通过针对不同用户和研究社区的三种交付成果来应对研究和社会挑战：1）修改和扩展人工智能方法以及公开共享的ASL数据和工具，以涵盖口语。虽然面部表情和头部姿势对手语的语法至关重要，在言语中也起着重要作用，但人们对这一点还没有很好的理解，因为还没有得到该项目开发的这类资源。新的数据和分析将为比较研究面部表情在各种形式中的作用以及面部表情在手语与口语中的作用打开大门。共享的原始数据、分析和可视化将为语言学和计算机科学研究开辟新的途径，研究非手动表达与语音和手势的时空同步作用。2)一个应用程序，以帮助美国手语学习者产生面部表情和头部姿势，以传达语法信息的手语;以及3）开发用于ASL视频的实时匿名化的工具，以保存非手动表达的语法信息，而德-该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的评估被认为是值得支持的。影响审查标准。

项目成果

Isolated Sign Recognition using ASL Datasets with Consistent Text-based Gloss Labeling and Curriculum Learning

• 发表时间: 2022
• 作者: Konstantinos M. Dafnis;Evgenia Chroni;C. Neidle;Dimitris N. Metaxas

ASL Video Corpora & Sign Bank: Resources Available through the American Sign Language Linguistic Research Project (ASLLRP)

• 发表时间: 2022-01
• 期刊: ArXiv
• 作者: C. Neidle;Augustine Opoku;Dimitris N. Metaxas

American Sign Language Video Anonymization to Support Online Participation of Deaf and Hard of Hearing Users

• DOI: 10.1145/3441852.3471200
• 发表时间: 2021-10
• 期刊: Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility
• 作者: Sooyeon Lee;Abraham Glasser;Becca Dingman;Zhaoyang Xia;Dimitris N. Metaxas;C. Neidle;Matt Huenerfauth

Understanding ASL Learners’ Preferences for a Sign Language Recording and Automatic Feedback System to Support Self-Study

了解 ASL 学习者对支持自学的手语录音和自动反馈系统的偏好

• DOI: 10.1145/3517428.3550367
• 发表时间: 2022
• 期刊: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility
• 作者: Hassan, Saad;Lee, Sooyeon;Metaxas, Dimitris;Neidle, Carol;Huenerfauth, Matt
• 通讯作者: Huenerfauth, Matt

Resources for Computer-Based Sign Recognition from Video, and the Criticality of Consistency of Gloss Labeling across Multiple Large ASL Video Corpora

• 发表时间: 2022
• 作者: C. Neidle;Augustine Opoku;Carey M. Ballard;Konstantinos M. Dafnis;Evgenia Chroni;Dimitris N. Metaxas