Telesign: Towards a One-Way American Sign Language Translator

PI假设聋人和听力健全社区之间的交流可以通过使用移动单向手语翻译器来增强。他将这种翻译器的初步原型命名为Telesign，它使用了一个可穿戴的电脑，一个安装在帽子边缘的扬声器，一个用于用户反馈和编辑的平视免提微型显示器，以及各种传感器（例如，嵌入用户帽子的摄像头，戴在腕带上的无线加速度计，以及Acceleglove加速度计/电位器系统）。在这个项目中，他的第一步将是比较聋人与听力正常的人（非手语）之间不同的交流方法，包括写作（目前的做法）、在移动设备上打字、手势驱动的手语单向翻译器和短语级单向翻译器，这两种方法都可以为现有的交流技术所获得的表现提供衡量标准，并为未来对Telesign的评估建立基线。这些方法将通过速度（每秒传达的命题和完成任务的时间）、感知工作量（NASA任务负载指数）和英语生成的清晰度来评估。然后，PI将显著扩展Telesign原型，将语音识别技术应用于手语识别，通过采用他的乔治亚理工学院手势工具包（GT2K），这是一个公开可用的隐马尔可夫模型（HMM）工具包，用于手势识别，用于这些任务。PI还将研究传感技术，例如在移动环境中对手的视觉跟踪，以便为长时间的符号翻译测试创建合理的原型。最后，PI将定义研究级任务，这些任务可用于评估聋人计算机增强通信辅助设备的性能（包括现有技术和新改进的远程通信系统）。更广泛的影响：与听力社区的日常交流对数百万听力损失患者构成了重大挑战。大多数听力正常的人不懂手语，对耳聋也知之甚少。这个项目试图通过研究语言生产的辅助工具来降低聋人与听力正常人群之间的沟通障碍。然而，日常使用一种成功的交流工具也可以帮助手语者更多地了解书面和口语的结构，就像日常互动帮助学生学习外语一样。这项研究将佐治亚理工学院的学生与亚特兰大地区聋人学校（一所K-12公立学校）的顾问和研究对象结合在一起，将一些全国顶尖的工程师和科学家展示给聋人社区，反之亦然。研究小组将积极招募聋人学者和聋人社区的母语手语者作为研究小组的成员，这也将有助于研究变得可行时的传播。

The PI hypothesizes that communication between the deaf and hearing communities can be augmented through the use of a mobile one-way sign language translator. His preliminary prototype of such a translator, dubbed Telesign, uses a body-worn computer, a speaker in the brim of the hat, a head-up, hands-free micro-display for user feedback and editing, and a variety of sensors (e.g., a camera embedded in the user's hat, wireless accelerometers worn in wristbands, and the Acceleglove accelerometer/potentiometer system). His first step in this project will be to compare different methods of communication between a deaf and hearing (non-signing) individual - including writing (current practice), typing on mobile devices, a gesture driven sign-by-sign one-way translator, and a phrase-level one-way translator - both to provide a measure for the performance obtained with existing communication technologies and to establish the baseline for future evaluation of Telesign. These methods will be evaluated by speed (propositions conveyed per second and time to task completion), perceived workload (NASA Task Load Index), and clarity of English generation. The PI will then significantly extend the Telesign prototype, by bringing speech recognition techniques to bear on sign language recognition by adapting his Georgia Tech Gesture Toolkit (GT2K), a publicly available hidden Markov model (HMM) toolkit for gesture recognition, for these tasks. The PI will also research sensing techniques, such as visual tracking of the hands in a mobile environment, in order to create reasonable prototypes for sign translation testing over an extended time. Finally, the PI will define research-grade tasks that can be used to evaluate the performance of computer enhanced communication aids for the deaf (both existing techniques and the new improved Telesign system).Broader Impacts: Everyday communication with the hearing community poses a major challenge to the millions of people with hearing loss. Most hearing people do not know sign language, and know very little about deafness in general. This project attempts to lower the communication barrier between the deaf and hearing populations, by researching aids for language production. However, everyday use of a successful communication aid may also help a signer learn more about the structure of written and spoken English, in much the same way that everyday interactions help students learn a foreign language. This research couples Georgia Tech students with consultants and subjects from the Atlanta Area School for the Deaf (a K-12 state school), exposing some of the nation's top student engineers and scientists to the deaf community, and vice versa. Deaf academics and native signers from the deaf community will be recruited actively as members or the research team, which will also help disseminate the research as it becomes viable.