NSF Convergence Accelerator Track H: AI-based Tools to Enhance Access and Opportunities for the Deaf

NSF 融合加速器轨道 H：基于人工智能的工具，增强聋人的获取和机会

• 批准号: 2235405
• 负责人: Dimitris Metaxas
• 金额: $ 75万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-12-15 至 2024-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2235405&HistoricalAwards=false
• 关键词: NSF; Convergence; Accelerator; Track; AI

We propose to develop sustainable, robust AI methods to overcome obstacles to digital communication and information access faced by Deaf and Hard-of-Hearing (DHH) individuals, empowering them personally and professionally. Users of American Sign Language (ASL), which has no standard written form, lack parity with hearing users in the digital arena. The proposed tools for privacy protection for ASL video communication and video search-by-example for access to multimedia digital resources build on prior NSF-funded AI research on linguistically-informed computer-based analysis and recognition of ASL from videos.PROBLEM #1. ASL signers cannot communicate anonymously about sensitive topics through videos in their native language; this is perceived by the Deaf community to be a serious problem.PROBLEM #2. There is no good way to look up a sign in a dictionary. Many ASL dictionaries enable sign look-up based on English translations, but what if the user does not understand the sign, or does not know its English translation? Others allow for search based on properties of ASL signs (e.g., handshape, location, movement type), but this is cumbersome, and a user must often look through hundreds of pictures of signs to find a target sign (if it is present at all in that dictionary).The tools to be developed will enable signers to anonymize ASL videos while preserving essential linguistic information conveyed by hands, arms, facial expressions, and head movements; and enable searching for a sign based on ASL input from a webcam or a video clip.Participants include DHH individuals, Deaf-owned companies, and members of other underrepresented minorities. The products will serve the 500,000 US signers and could be extended to other sign languages. The proposed application development brings together state-of-the-art research on: (1) video anonymization (using an asymmetric encoder-decoder structured image generator to generate high-resolution target frames driven by the original signing from the low-resolution source frames for anonymization, based on optical flow and confidence maps); (2) computer-based sign recognition from video (bidirectional skeleton-based isolated sign recognition using Graph Convolution Networks); and (3) HCI, including DHH user studies to assess desiderata for user interfaces for the proposed applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

我们建议开发可持续、强大的人工智能方法，以克服聋人和重听人(DHH)面临的数字通信和信息访问障碍，赋予他们个人和专业能力。美国手语(ASL)的用户没有标准的书面形式，在数字舞台上缺乏与听力用户的平等。建议的ASL视频通信和视频搜索的隐私保护工具，用于访问多媒体数字资源，建立在NSF资助的先前人工智能研究的基础上，该研究基于计算机对视频中ASL的语言信息进行分析和识别。问题1.ASL签名者不能通过他们的母语视频匿名交流敏感话题；这被聋人社区认为是一个严重的问题。问题2。没有好的方法在词典中查找手势。许多ASL词典支持基于英文翻译的手语查找，但如果用户不理解手语，或不知道其英文翻译怎么办？其他工具则允许基于ASL手势的属性(例如，手形、位置、移动类型)进行搜索，但这很麻烦，用户通常必须查看数百张手势的图片才能找到目标手势(如果该词典中存在目标手势)。将要开发的工具将使签名者能够匿名ASL视频，同时保留通过手、手臂、面部表情和头部动作传达的基本语言信息；并能够基于从网络摄像头或视频剪辑输入的ASL搜索手势。参与者包括DHH个人、聋人所有的公司和其他未被充分代表的少数族裔成员。这些产品将服务于50万美国手语用户，并可能扩展到其他手语。提出的应用开发结合了以下方面的最新研究：(1)视频匿名化(使用非对称编解码器结构化图像生成器，基于光流和置信度图，从用于匿名化的低分辨率源帧生成由原始签名驱动的高分辨率目标帧)；(2)基于计算机的视频标志识别(基于双向骨架的基于图形卷积网络的孤立标志识别)；和(3)人机界面，包括DHH用户研究，以评估拟议应用程序的用户界面的期望数据。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Exploring the Design Space of Automatically Generated Emotive Captions for Deaf or Hard of Hearing Users

探索为聋哑或听力障碍用户自动生成情感字幕的设计空间

• DOI: 10.1145/3544549.3585880
• 发表时间: 2023
• 期刊: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA '23
• 作者: Hassan, Saad;Ding, Yao;Kerure, Agneya Abhimanyu;Miller, Christi;Burnett, John;Biondo, Emily;Gilbert, Brenden
• 通讯作者: Gilbert, Brenden

DiffSLVA: Harnessing Diffusion Models for Sign Language Video Anonymization

• DOI: 10.48550/arxiv.2311.16060
• 发表时间: 2023-11
• 期刊: ArXiv
• 作者: Zhaoyang Xia;C. Neidle;Dimitris N. Metaxas

Modeling Word Importance in Conversational Transcripts: Toward improved live captioning for Deaf and hard of hearing viewers

• DOI: 10.1145/3587281.3587290
• 发表时间: 2023-04
• 期刊: Proceedings of the 20th International Web for All Conference
• 作者: Akhter Al Amin;Saad Hassan;Matt Huenerfauth;Cecilia Ovesdotter Alm

Challenges for Linguistically-Driven Computer-Based Sign Recognition from Continuous Signing for American Sign Language

美国手语连续手语对语言驱动的基于计算机的手语识别的挑战

• 发表时间: 2023
• 期刊: arXiv.org
• 作者: Neidle, Carol

Understanding How Deaf and Hard of Hearing Viewers Visually Explore Captioned Live TV News

了解失聪和听力障碍观众如何视觉探索带字幕的直播电视新闻

• DOI: 10.1145/3587281.3587287
• 发表时间: 2023
• 期刊: Proceedings of the 20th International Web for All Conference
• 作者: Amin, Akhter Al;Hassan, Saad;Lee, Sooyeon;Huenerfauth, Matt
• 通讯作者: Huenerfauth, Matt