Telesign: Towards a One-Way American Sign Language Translator

PI假设聋人和听力社区之间的沟通可以通过使用移动的单向手语翻译器来增强。 他的这种翻译器的初步原型，被称为Telesign，使用一个身体佩戴的计算机，一个在帽子边缘的扬声器，一个用于用户反馈和编辑的头戴式免提微型显示器，以及各种传感器（例如，嵌入用户帽子中的照相机、佩戴在腕带中的无线加速度计、以及Acceleglove加速度计/电位计系统）。 他在这个项目中的第一步将是比较聋人和听力正常者之间不同的交流方法（非签字）个人-包括书面（当前实践），在移动的设备上打字，手势驱动的逐符号单向翻译器，和一个短语级单向翻译器既为现有通信技术的性能提供衡量标准，又为未来评估电报 这些方法将通过速度（每秒传达的命题和完成任务的时间），感知工作量（NASA任务负荷指数）和英语生成的清晰度进行评估。 然后，PI将大大扩展Telesign原型，通过调整他的格鲁吉亚技术手势工具包（GT 2K），一个公开的隐马尔可夫模型（HMM）工具包，用于手势识别，将语音识别技术应用于手语识别。 PI还将研究传感技术，例如在移动的环境中对手部进行视觉跟踪，以便在较长时间内为符号翻译测试创建合理的原型。 最后，PI将定义研究级的任务，可用于评估计算机增强的聋人交流辅助设备（包括现有技术和新改进的Telesign系统）的性能。更广泛的影响：与听力社区的日常交流对数百万听力损失患者构成了重大挑战。 大多数听力正常的人不懂手语，对耳聋的一般情况也知之甚少。 该项目试图通过研究语言产生的辅助工具来降低聋人和听力正常人群之间的沟通障碍。 然而，日常使用一个成功的沟通辅助工具也可以帮助签名者更多地了解书面和口语英语的结构，就像日常互动帮助学生学习外语一样。 这项研究将格鲁吉亚理工学院的学生与亚特兰大地区聋人学校（一所K-12州立学校）的顾问和受试者联系起来，将一些全国顶尖的工程师和科学家学生暴露给聋人社区，反之亦然。 聋人学者和聋人社区的本地签名者将被积极招募为研究团队的成员，这也将有助于传播研究，因为它变得可行。

The PI hypothesizes that communication between the deaf and hearing communities can be augmented through the use of a mobile one-way sign language translator. His preliminary prototype of such a translator, dubbed Telesign, uses a body-worn computer, a speaker in the brim of the hat, a head-up, hands-free micro-display for user feedback and editing, and a variety of sensors (e.g., a camera embedded in the user's hat, wireless accelerometers worn in wristbands, and the Acceleglove accelerometer/potentiometer system). His first step in this project will be to compare different methods of communication between a deaf and hearing (non-signing) individual - including writing (current practice), typing on mobile devices, a gesture driven sign-by-sign one-way translator, and a phrase-level one-way translator - both to provide a measure for the performance obtained with existing communication technologies and to establish the baseline for future evaluation of Telesign. These methods will be evaluated by speed (propositions conveyed per second and time to task completion), perceived workload (NASA Task Load Index), and clarity of English generation. The PI will then significantly extend the Telesign prototype, by bringing speech recognition techniques to bear on sign language recognition by adapting his Georgia Tech Gesture Toolkit (GT2K), a publicly available hidden Markov model (HMM) toolkit for gesture recognition, for these tasks. The PI will also research sensing techniques, such as visual tracking of the hands in a mobile environment, in order to create reasonable prototypes for sign translation testing over an extended time. Finally, the PI will define research-grade tasks that can be used to evaluate the performance of computer enhanced communication aids for the deaf (both existing techniques and the new improved Telesign system).Broader Impacts: Everyday communication with the hearing community poses a major challenge to the millions of people with hearing loss. Most hearing people do not know sign language, and know very little about deafness in general. This project attempts to lower the communication barrier between the deaf and hearing populations, by researching aids for language production. However, everyday use of a successful communication aid may also help a signer learn more about the structure of written and spoken English, in much the same way that everyday interactions help students learn a foreign language. This research couples Georgia Tech students with consultants and subjects from the Atlanta Area School for the Deaf (a K-12 state school), exposing some of the nation's top student engineers and scientists to the deaf community, and vice versa. Deaf academics and native signers from the deaf community will be recruited actively as members or the research team, which will also help disseminate the research as it becomes viable.