HCC: Small: Collaborative Research: Real-Time Captioning by Groups of Non-Experts for Deaf and Hard of Hearing Students

许多聋哑和重听学生使用实时字幕参与教育。通常，实时字幕是由熟练的专业字幕员（速记员）提供的，他们使用专门的键盘或软件来保持每分钟225个单词的自然语速。但专业的资本主义者是昂贵的，必须提前安排至少一个小时的时间。自动语音识别（ASR）正在进步，但在实际教室中仍然存在较高的错误率。在这项涉及罗切斯特大学和罗切斯特理工学院的合作努力中，pi将通过混合人力和机器驱动的字幕来解决这些问题，以低成本实时按需制作字幕。pi的方法是让多个非专家和ASR在5秒内共同为语音配上字幕，借助鼓励快速，不完整的实时音频字幕的接口。由于非专家无法跟上自然语速，新的算法将实时合并不完整的字幕。（虽然序列对齐问题可以用动态规划精确地解决，但现有的方法速度太慢，对输入错误的鲁棒性不够，而且没有结合自然语言语义。）系统地改变声音的显著性将鼓励完全覆盖讲话。非专业字幕将实时训练ASR引擎，以便在讲座期间提高ASR。（传统的ASR训练方法假设训练是离线进行的。）quikCaption移动应用程序将体现这些想法，并将通过设计会议、实验室研究和课堂部署，与国家聋人技术研究所（NTID）的聋人和重听学生一起迭代设计。非专业的captionists可以从广泛的来源：志愿者愿意贡献自己的时间，同学有相关的领域知识，或随时可用的有偿工人。它们可能是本地的（在教室里），也可能是远程的。Captionists可能有经验，从以前的快速说明会议，或从现有的市场（如机械土耳其）的需求招募的新手群众工人。灵活的工人池将允许以低成本和只在需要时提供实时字幕。更广泛的影响：这项研究将极大地改善聋哑和重听学生的教育，使他们能够获得偶然的机会，例如课后对话或最后一刻没有安排翻译或字幕的客座讲座。实时字幕在学校课程、艺术表演和政治活动等其他场合也很有用。年长的有听力障碍的成年人通常更喜欢字幕，这一群体的人数相当可观，而且还在不断增长；听力正常的人可能会受益，因为字幕是语音自动翻译的第一步。作为该项目的一部分，为实时合并不完整的自然语言而开发的算法可能适用于其他应用，如协作翻译或在嘈杂媒体上的通信。

Many deaf and hard of hearing students use real-time captioning to participate in education. Generally, real-time captions are provided by skilled professional captionists (stenographers) who use specialized keyboards or software to keep up with natural speaking rates of up to 225 words per minute. But professional captionists are expensive and must be arranged in advance in blocks of at least an hour. Automatic speech recognition (ASR) is improving, but still experiences high error rates in real classrooms. In this collaborative effort involving the University of Rochester and Rochester Institute of Technology, the PIs will address these issues by blending human- and machine-powered captioning to produce captions on demand, in real time, for low cost. The PIs' approach is for multiple non-experts and ASR to collectively caption speech in under 5 seconds, with the help of interfaces which encourage quick, incomplete captioning of live audio. Because non-experts cannot keep up with natural speaking rates, new algorithms will merge incomplete captions in real time. (While the sequence alignment problem can be solved exactly with dynamic programming, existing approaches are too slow, are not robust to input error, and do not incorporate natural language semantics.) Systematically varying audio saliency will encourage complete coverage of speech. Non-expert captions will train ASR engines in real time, so that ASR may improve during a lecture. (Traditional approaches for ASR training assume that training occurs offline.) The quikCaption mobile application will embody these ideas and will be iteratively designed with deaf and hard of hearing students at the National Technical Institute of the Deaf (NTID) via design sessions, lab studies and in-class deployments. Non-expert captionists can be drawn from broad sources: volunteers willing to donate their time, classmates with relevant domain knowledge, or always-available paid workers. They may be local (in the classroom) or remote. Captionists may have experience from prior quikCaption sessions, or novice crowd workers recruited on demand from existing marketplaces (e.g., Mechanical Turk). A flexible worker pool will allow real-time captions to be available on demand at low cost and for only as long as needed.Broader Impacts: This research will dramatically improve education for deaf and hard of hearing students by enabling access to serendipitous opportunities, such as conversations after class or last-minute guest lectures for which no interpreter or captionist was arranged. Real-time captioning will also be useful in other settings such as school programs, artistic performances, and political events. Older hard of hearing adults usually prefer captioning, and represent a sizable and growing population; hearing people may benefit because captioning is a first step in automatic translation of aural speech. The algorithms developed as part of this project for real-time merging of incomplete natural language will likely be adaptable for other applications such as collaborative translation or communication over noisy mediums.