NSF Convergence Accelerator Track H: AI-based Tools to Enhance Access and Opportunities for the Deaf

我们建议开发可持续的、强大的人工智能方法，以克服聋人和听障人士（DHH）在数字通信和信息获取方面面临的障碍，赋予他们个人和专业能力。美国手语（ASL）没有标准的书面形式，在数字领域与听力健全的用户缺乏平等。所提出的用于ASL视频通信的隐私保护工具和用于访问多媒体数字资源的视频示例搜索工具建立在先前nsf资助的基于语言的计算机分析和识别视频中ASL的人工智能研究的基础上。问题# 1。手语使用者不能通过母语视频匿名交流敏感话题；聋人社区认为这是一个严重的问题。问题# 2。在字典中查找标志没有好的方法。许多ASL字典支持基于英语翻译的符号查找，但是如果用户不理解该符号，或者不知道其英语翻译怎么办？还有一些基于手语特征的搜索（例如，手型、位置、移动类型），但这很麻烦，用户必须经常查看数百张手语图片才能找到目标手势（如果字典中有的话）。待开发的工具将使签名者能够匿名化美国手语视频，同时保留由手、手臂、面部表情和头部动作传达的基本语言信息；并且可以根据来自网络摄像头或视频剪辑的ASL输入来搜索标志。参与者包括DHH个人、聋人所有的公司和其他代表性不足的少数民族成员。这些产品将为50万美国手语使用者提供服务，并可以扩展到其他手语。提出的应用程序开发汇集了最先进的研究：(1)视频匿名化（基于光流和置信度图，使用非对称编码器-解码器结构化图像生成器从用于匿名化的低分辨率源帧生成由原始签名驱动的高分辨率目标帧）；(2)基于视频的计算机符号识别（基于图卷积网络的双向骨架隔离符号识别）；(3)人机交互，包括DHH用户研究，以评估拟议应用程序的用户界面需求。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

We propose to develop sustainable, robust AI methods to overcome obstacles to digital communication and information access faced by Deaf and Hard-of-Hearing (DHH) individuals, empowering them personally and professionally. Users of American Sign Language (ASL), which has no standard written form, lack parity with hearing users in the digital arena. The proposed tools for privacy protection for ASL video communication and video search-by-example for access to multimedia digital resources build on prior NSF-funded AI research on linguistically-informed computer-based analysis and recognition of ASL from videos.PROBLEM #1. ASL signers cannot communicate anonymously about sensitive topics through videos in their native language; this is perceived by the Deaf community to be a serious problem.PROBLEM #2. There is no good way to look up a sign in a dictionary. Many ASL dictionaries enable sign look-up based on English translations, but what if the user does not understand the sign, or does not know its English translation? Others allow for search based on properties of ASL signs (e.g., handshape, location, movement type), but this is cumbersome, and a user must often look through hundreds of pictures of signs to find a target sign (if it is present at all in that dictionary).The tools to be developed will enable signers to anonymize ASL videos while preserving essential linguistic information conveyed by hands, arms, facial expressions, and head movements; and enable searching for a sign based on ASL input from a webcam or a video clip.Participants include DHH individuals, Deaf-owned companies, and members of other underrepresented minorities. The products will serve the 500,000 US signers and could be extended to other sign languages. The proposed application development brings together state-of-the-art research on: (1) video anonymization (using an asymmetric encoder-decoder structured image generator to generate high-resolution target frames driven by the original signing from the low-resolution source frames for anonymization, based on optical flow and confidence maps); (2) computer-based sign recognition from video (bidirectional skeleton-based isolated sign recognition using Graph Convolution Networks); and (3) HCI, including DHH user studies to assess desiderata for user interfaces for the proposed applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.