SBIR Phase I: Real-Time Artificial Intelligence (AI) Bidirectional American Sign Language (ASL) Communication System

这个小企业创新研究（SBIR）第一阶段项目的更广泛影响是通过自动手语识别改善聋人和重听人（D/HH）个人和听力社区之间的沟通。仅在美国就有超过4800万D/HH个人，他们总共拥有870亿美元的购买力。企业似乎没有为这个群体提供充分的服务，大量针对众多公司的《美国残疾人法案》（ADA）诉讼证明了这一点。拟议的技术将为组织提供即插即用的软件，以改善他们与D/HH个人的互动。在没有口译员的情况下，企业和政府将能够与其D/HH员工、客户或选民进行互动。这项技术可以集成到各种平台中，从零售销售点设备到聊天机器人和视频/电话会议系统。这项小型企业创新研究（SBIR）第一阶段项目旨在开发执行无约束手语识别和自然手语生产的技术。具体而言，由于缺乏该领域的训练数据，目前训练语言翻译模型的方法不适合处理手语领域。此外，目前所有用于生成美国手语（ASL）的方法（除了动作捕捉，这是不可扩展的）都会导致虚拟角色做出做作、不自然的手势。该项目将通过半监督专家增强模型和数据增强技术，为ASL领域内的这些问题开发解决方案。技术障碍包括缺乏处理高维低资源语言域的模型，以及缺乏足够大的数据集。技术里程碑包括创建半监督数据集、工程数据增强技术、生成自然签名化身以及执行广泛的可用性测试。该项目旨在创造一种在低资源手语和英语之间自动翻译的方法，以提高聋人和重听社区的可及性和公平性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact of this Small Business Innovation Research (SBIR) Phase I project is to improve the communication between Deaf and Hard of Hearing (D/HH) individuals and the hearing community through automated sign language recognition. In the United States alone there are over 48 million D/HH individuals, who in total possess $87 billion in purchasing power. It appears businesses are not adequately serving this community, as is evidenced by the plethora of Americans with Disabilities Act (ADA) lawsuits against numerous companies. The proposed technology will provide plug-and-play software for organizations to improve their interactions with D/HH individuals. Businesses and governments will be able to interact with their D/HH employees, customers, or constituents when interpreters are unavailable. This technology can be integrated into a variety of platforms, from retail point-of-sale equipment to chatbots and video/teleconferencing systems.This Small Business Innovation Research (SBIR) Phase 1 project aims to develop technology to perform unconstrained sign language recognition and natural sign language production. Specifically, current methods to train language translation models are ill-equipped to handle the sign language domain due to the lack of training data within this domain. Additionally, all currently established methods (apart from motion capture, which is unscalable) for producing American Sign Language (ASL) result in stilted, unnatural signing from an avatar. This project will develop solutions to these issues within the domain of ASL via semi-supervised expert-augmented models and data augmentation techniques. Technical hurdles include the lack of models to handle high-dimensional low-resource language domains, and lack of sufficiently large datasets. Technical milestones include creating semi-supervised datasets, engineering data augmentation techniques, generating a natural signing avatar, and performing extensive usability testing. This project aims to produce a method for automatically interpreting between a low-resource sign language and English to improve accessibility and increase equity for the Deaf and Hard of Hearing communities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.