HCC: Small: Collaborative Research: Real-Time Captioning by Groups of Non-Experts for Deaf and Hard of Hearing Students

许多聋哑生使用实时字幕参与教育。一般来说，实时字幕是由熟练的专业字幕员(速记员)提供的，他们使用专门的键盘或软件来跟上高达每分钟225字的自然发音速度。但专业的字幕制作费用很高，而且必须提前安排至少一个小时的字幕。自动语音识别(ASR)正在改进，但在真实课堂上仍有很高的错误率。在这项涉及罗切斯特大学和罗切斯特理工学院的合作努力中，PI将通过混合人工和机器驱动的字幕来解决这些问题，以低成本实时按需制作字幕。PIS的方法是让多个非专家和ASR在5秒内集体为语音添加字幕，并借助鼓励对现场音频进行快速、不完整的字幕的界面。由于非专业人士无法跟上自然发音速度，新的算法将实时合并不完整的字幕。(虽然序列比对问题可以用动态编程准确地解决，但现有的方法太慢，对输入错误不健壮，并且没有结合自然语言语义。)系统地改变音频的显著程度将鼓励对语音的全面覆盖。非专家字幕将实时训练ASR引擎，这样ASR可能会在一次讲座中提高。(ASR培训的传统方法假定培训离线进行。)QukCaption移动应用程序将体现这些想法，并将通过设计会议、实验室研究和课堂部署，与国家聋人技术学院(NTID)的聋人和重听学生反复设计。非专业的字幕作者可以来自广泛的来源：愿意贡献时间的志愿者，具有相关领域知识的同学，或者总是有空的付费工人。他们可能是本地的(在教室里)，也可能是远程的。Captionist可能从之前的QukCaption会议中获得经验，或者从现有市场(例如机械土耳其人)按需招募新手群体工作人员。灵活的工作人员池将允许以低成本按需提供实时字幕，并且只在需要的时间内提供。广泛影响：这项研究将极大地改善聋哑人和重听学生的教育，使他们能够获得偶然的机会，例如课后对话或没有安排翻译或字幕演讲者的最后一刻的客座讲座。实时字幕在其他设置中也很有用，例如学校节目、艺术表演和政治活动。年长的重听成年人通常更喜欢字幕，这代表了一个相当大的且不断增长的人口；听力正常的人可能会受益，因为字幕是听觉语音自动翻译的第一步。作为这个项目的一部分，为实时合并不完整的自然语言而开发的算法很可能适用于其他应用，如协作翻译或在嘈杂媒体上的交流。

Many deaf and hard of hearing students use real-time captioning to participate in education. Generally, real-time captions are provided by skilled professional captionists (stenographers) who use specialized keyboards or software to keep up with natural speaking rates of up to 225 words per minute. But professional captionists are expensive and must be arranged in advance in blocks of at least an hour. Automatic speech recognition (ASR) is improving, but still experiences high error rates in real classrooms. In this collaborative effort involving the University of Rochester and Rochester Institute of Technology, the PIs will address these issues by blending human- and machine-powered captioning to produce captions on demand, in real time, for low cost. The PIs' approach is for multiple non-experts and ASR to collectively caption speech in under 5 seconds, with the help of interfaces which encourage quick, incomplete captioning of live audio. Because non-experts cannot keep up with natural speaking rates, new algorithms will merge incomplete captions in real time. (While the sequence alignment problem can be solved exactly with dynamic programming, existing approaches are too slow, are not robust to input error, and do not incorporate natural language semantics.) Systematically varying audio saliency will encourage complete coverage of speech. Non-expert captions will train ASR engines in real time, so that ASR may improve during a lecture. (Traditional approaches for ASR training assume that training occurs offline.) The quikCaption mobile application will embody these ideas and will be iteratively designed with deaf and hard of hearing students at the National Technical Institute of the Deaf (NTID) via design sessions, lab studies and in-class deployments. Non-expert captionists can be drawn from broad sources: volunteers willing to donate their time, classmates with relevant domain knowledge, or always-available paid workers. They may be local (in the classroom) or remote. Captionists may have experience from prior quikCaption sessions, or novice crowd workers recruited on demand from existing marketplaces (e.g., Mechanical Turk). A flexible worker pool will allow real-time captions to be available on demand at low cost and for only as long as needed.Broader Impacts: This research will dramatically improve education for deaf and hard of hearing students by enabling access to serendipitous opportunities, such as conversations after class or last-minute guest lectures for which no interpreter or captionist was arranged. Real-time captioning will also be useful in other settings such as school programs, artistic performances, and political events. Older hard of hearing adults usually prefer captioning, and represent a sizable and growing population; hearing people may benefit because captioning is a first step in automatic translation of aural speech. The algorithms developed as part of this project for real-time merging of incomplete natural language will likely be adaptable for other applications such as collaborative translation or communication over noisy mediums.