NSF Convergence Accelerator Track H: AI-based Tools to Enhance Access and Opportunities for the Deaf

NSF 融合加速器轨道 H：基于人工智能的工具，增强聋人的获取和机会

• 批准号: 2235405
• 负责人: Dimitris Metaxas
• 金额: $ 75万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-12-15 至 2024-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2235405&HistoricalAwards=false
• 关键词: NSF; Convergence; Accelerator; Track; AI

We propose to develop sustainable, robust AI methods to overcome obstacles to digital communication and information access faced by Deaf and Hard-of-Hearing (DHH) individuals, empowering them personally and professionally. Users of American Sign Language (ASL), which has no standard written form, lack parity with hearing users in the digital arena. The proposed tools for privacy protection for ASL video communication and video search-by-example for access to multimedia digital resources build on prior NSF-funded AI research on linguistically-informed computer-based analysis and recognition of ASL from videos.PROBLEM #1. ASL signers cannot communicate anonymously about sensitive topics through videos in their native language; this is perceived by the Deaf community to be a serious problem.PROBLEM #2. There is no good way to look up a sign in a dictionary. Many ASL dictionaries enable sign look-up based on English translations, but what if the user does not understand the sign, or does not know its English translation? Others allow for search based on properties of ASL signs (e.g., handshape, location, movement type), but this is cumbersome, and a user must often look through hundreds of pictures of signs to find a target sign (if it is present at all in that dictionary).The tools to be developed will enable signers to anonymize ASL videos while preserving essential linguistic information conveyed by hands, arms, facial expressions, and head movements; and enable searching for a sign based on ASL input from a webcam or a video clip.Participants include DHH individuals, Deaf-owned companies, and members of other underrepresented minorities. The products will serve the 500,000 US signers and could be extended to other sign languages. The proposed application development brings together state-of-the-art research on: (1) video anonymization (using an asymmetric encoder-decoder structured image generator to generate high-resolution target frames driven by the original signing from the low-resolution source frames for anonymization, based on optical flow and confidence maps); (2) computer-based sign recognition from video (bidirectional skeleton-based isolated sign recognition using Graph Convolution Networks); and (3) HCI, including DHH user studies to assess desiderata for user interfaces for the proposed applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

我们建议开发可持续的，强大的人工智能方法，以克服聋人和听力困难（DHH）个人面临的数字通信和信息访问障碍，增强他们的个人和专业能力。美国手语（ASL）没有标准的书面形式，其用户在数字竞技场中与听力用户缺乏平等。针对ASL视频通信和用于访问多媒体数字资源的视频搜索的隐私保护的所提出的工具建立在先前NSF资助的关于从视频中对ASL进行基于语言的基于计算机的分析和识别的AI研究的基础上。手语签名者无法通过母语视频匿名交流敏感话题;聋人社区认为这是一个严重的问题。问题2。在字典里查一个标志没有好办法。许多美国手语词典支持基于英语翻译的符号查找，但是如果用户不理解符号，或者不知道它的英语翻译呢？ 其他允许基于ASL标志的属性进行搜索（例如，手形、位置、运动类型），但是这是麻烦的，并且用户必须经常浏览数百张标志的图片以找到目标标志将开发的工具将使签名者能够对美国手语视频进行匿名处理，同时保留由手、手臂、面部表情和头部运动传达的基本语言信息;并允许基于来自网络摄像头或视频剪辑的ASL输入搜索标志。参与者包括DHH个人，聋人拥有的公司和其他代表性不足的少数民族成员。这些产品将为50万美国手语用户提供服务，并可能扩展到其他手语。拟议的应用程序开发汇集了最先进的研究：（1）视频匿名化（使用非对称编码器-解码器结构化图像生成器来基于光流和置信图从低分辨率源帧生成由原始签名驱动的高分辨率目标帧以用于匿名化）;（2）基于计算机的视频标志识别（使用图卷积网络的基于双向卷积子的孤立符号识别）;以及（3）HCI，包括DHH用户研究，以评估拟议应用程序的用户界面的必要性。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Exploring the Design Space of Automatically Generated Emotive Captions for Deaf or Hard of Hearing Users

探索为聋哑或听力障碍用户自动生成情感字幕的设计空间

• DOI: 10.1145/3544549.3585880
• 发表时间: 2023
• 期刊: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA '23
• 作者: Hassan, Saad;Ding, Yao;Kerure, Agneya Abhimanyu;Miller, Christi;Burnett, John;Biondo, Emily;Gilbert, Brenden
• 通讯作者: Gilbert, Brenden

DiffSLVA: Harnessing Diffusion Models for Sign Language Video Anonymization

• DOI: 10.48550/arxiv.2311.16060
• 发表时间: 2023-11
• 期刊: ArXiv
• 作者: Zhaoyang Xia;C. Neidle;Dimitris N. Metaxas

Modeling Word Importance in Conversational Transcripts: Toward improved live captioning for Deaf and hard of hearing viewers

• DOI: 10.1145/3587281.3587290
• 发表时间: 2023-04
• 期刊: Proceedings of the 20th International Web for All Conference
• 作者: Akhter Al Amin;Saad Hassan;Matt Huenerfauth;Cecilia Ovesdotter Alm

Challenges for Linguistically-Driven Computer-Based Sign Recognition from Continuous Signing for American Sign Language

美国手语连续手语对语言驱动的基于计算机的手语识别的挑战

• 发表时间: 2023
• 期刊: arXiv.org
• 作者: Neidle, Carol

Understanding How Deaf and Hard of Hearing Viewers Visually Explore Captioned Live TV News

了解失聪和听力障碍观众如何视觉探索带字幕的直播电视新闻

• DOI: 10.1145/3587281.3587287
• 发表时间: 2023
• 期刊: Proceedings of the 20th International Web for All Conference
• 作者: Amin, Akhter Al;Hassan, Saad;Lee, Sooyeon;Huenerfauth, Matt
• 通讯作者: Huenerfauth, Matt