HCC: Small: Collaborative Research: Real-Time Captioning by Groups of Non-Experts for Deaf and Hard of Hearing Students

HCC：小型：协作研究：由非专家小组为聋哑和听力障碍学生提供实时字幕

• 批准号: 1218209
• 负责人: Jeffrey Bigham
• 金额: $ 41.92万
• 依托单位: University of Rochester
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-08-01 至 2014-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1218209&HistoricalAwards=false
• 关键词: HCC; Small; Collaborative; Research; Real

Many deaf and hard of hearing students use real-time captioning to participate in education. Generally, real-time captions are provided by skilled professional captionists (stenographers) who use specialized keyboards or software to keep up with natural speaking rates of up to 225 words per minute. But professional captionists are expensive and must be arranged in advance in blocks of at least an hour. Automatic speech recognition (ASR) is improving, but still experiences high error rates in real classrooms. In this collaborative effort involving the University of Rochester and Rochester Institute of Technology, the PIs will address these issues by blending human- and machine-powered captioning to produce captions on demand, in real time, for low cost. The PIs' approach is for multiple non-experts and ASR to collectively caption speech in under 5 seconds, with the help of interfaces which encourage quick, incomplete captioning of live audio. Because non-experts cannot keep up with natural speaking rates, new algorithms will merge incomplete captions in real time. (While the sequence alignment problem can be solved exactly with dynamic programming, existing approaches are too slow, are not robust to input error, and do not incorporate natural language semantics.) Systematically varying audio saliency will encourage complete coverage of speech. Non-expert captions will train ASR engines in real time, so that ASR may improve during a lecture. (Traditional approaches for ASR training assume that training occurs offline.) The quikCaption mobile application will embody these ideas and will be iteratively designed with deaf and hard of hearing students at the National Technical Institute of the Deaf (NTID) via design sessions, lab studies and in-class deployments. Non-expert captionists can be drawn from broad sources: volunteers willing to donate their time, classmates with relevant domain knowledge, or always-available paid workers. They may be local (in the classroom) or remote. Captionists may have experience from prior quikCaption sessions, or novice crowd workers recruited on demand from existing marketplaces (e.g., Mechanical Turk). A flexible worker pool will allow real-time captions to be available on demand at low cost and for only as long as needed.Broader Impacts: This research will dramatically improve education for deaf and hard of hearing students by enabling access to serendipitous opportunities, such as conversations after class or last-minute guest lectures for which no interpreter or captionist was arranged. Real-time captioning will also be useful in other settings such as school programs, artistic performances, and political events. Older hard of hearing adults usually prefer captioning, and represent a sizable and growing population; hearing people may benefit because captioning is a first step in automatic translation of aural speech. The algorithms developed as part of this project for real-time merging of incomplete natural language will likely be adaptable for other applications such as collaborative translation or communication over noisy mediums.

许多聋人和重听学生使用实时字幕参与教育。 一般来说，实时字幕由熟练的专业字幕员（速记员）提供，他们使用专门的键盘或软件来跟上每分钟高达225个单词的自然语速。 但是专业的字幕人员很贵，而且必须提前安排，至少一个小时。 自动语音识别（ASR）正在改进，但在真实的教室中仍然经历高错误率。 在这项涉及罗切斯特大学和罗切斯特理工学院的合作努力中，PI将通过混合人力和机器驱动的字幕来解决这些问题，以真实的时间和低成本按需制作字幕。 PI的方法是让多个非专家和ASR在5秒内共同为语音添加字幕，并借助界面鼓励对现场音频进行快速、不完整的字幕。 由于非专家无法跟上自然的说话速度，新的算法将在真实的时间合并不完整的字幕。（虽然序列比对问题可以用动态编程精确解决，但现有方法太慢，对输入错误不鲁棒，并且不包含自然语言语义。 系统地改变音频显著性将鼓励语音的完全覆盖。 非专家字幕将在真实的时间内训练ASR引擎，以便ASR可以在讲座期间改进。（ASR培训的传统方法假设培训是离线进行的。 quikCaption移动的应用程序将体现这些想法，并将通过设计会议、实验室研究和课堂部署，与国家聋人技术研究所（NTID）的聋人和重听学生一起迭代设计。 非专业的字幕可以从广泛的来源：志愿者愿意贡献自己的时间，同学与相关领域的知识，或随时可用的有偿工人。 他们可能是本地的（在教室里）或远程的。 字幕员可能有之前quikCaption会话的经验，或者根据现有市场的需求招募的新手人群工作者（例如，机械土耳其人）。 一个灵活的工作人员池将允许实时字幕以低成本按需提供，并且只在需要的时间内提供。更广泛的影响：这项研究将大大改善聋人和重听学生的教育，使他们能够获得偶然的机会，例如课后对话或最后一分钟的客座讲座，而这些讲座没有安排翻译或字幕。 实时字幕在其他场合也很有用，如学校节目、艺术表演和政治活动。 老年重听人通常更喜欢字幕，并代表了相当大的和不断增长的人口;听力的人可能会受益，因为字幕是听觉语音自动翻译的第一步。 作为该项目的一部分，为实时合并不完整的自然语言而开发的算法可能适用于其他应用，如协作翻译或通过嘈杂介质进行通信。