Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing
Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing
复制标题
评估自动生成的字幕对于聋哑人或听力障碍人士的可用性
DOI：
--
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Matt Huenerfauth
中科院分区：
文献类型：
--
作者：
Sushant Kafle;Matt Huenerfauth
自动语音识别（ASR）技术的准确性已经提高，但在许多环境中仍然不完善。评估ASR性能的研究人员通常专注于提高单词错误率（WER）指标，但在许多应用程序中，WER与人类受试者的性能几乎没有相关性。我们提出了一个新的以字幕为中心的评估指标，可以更好地预测ASR识别错误对聋人或听力困难者（DHH）自动生成字幕的可用性的影响。通过对30个DHH用户的用户研究，我们将我们的新指标与传统的WER指标在标题可用性评估任务上进行了比较。在对ASR文本输出（具有相同的WER）的并排比较中，DHH参与者更喜欢我们的新指标所偏好的文本。此外，我们的指标有显着更高的相关性与DHH参与者的主观评分的可用性的标题，相比WER指标和参与者的主观评分之间的相关性。这个新的指标可以用来选择ASR系统的字幕应用程序，它可能是一个更好的指标，为ASR研究人员在优化ASR系统时考虑。
The accuracy of Automated Speech Recognition (ASR) technology has improved, but it is still imperfect in many settings. Researchers who evaluate ASR performance often focus on improving the Word Error Rate (WER) metric, but WER has been found to have little correlation with human-subject performance on many applications. We propose a new captioning-focused evaluation metric that better predicts the impact of ASR recognition errors on the usability of automatically generated captions for people who are Deaf or Hard of Hearing (DHH). Through a user study with 30 DHH users, we compared our new metric with the traditional WER metric on a caption usability evaluation task. In a side-by-side comparison of pairs of ASR text output (with identical WER), the texts preferred by our new metric were preferred by DHH participants. Further, our metric had significantly higher correlation with DHH participants' subjective scores on the usability of a caption, as compared to the correlation between WER metric and participant subjective scores. This new metric could be used to select ASR systems for captioning applications, and it may be a better metric for ASR researchers to consider when optimizing ASR systems.
DOI：
10.1037/a0020990
发表时间：
2011-04
期刊：
Journal of experimental psychology. Human perception and performance
影响因子：
--
作者：
Rayner K;Slattery TJ;Drieghe D;Liversedge SP
通讯作者：
Liversedge SP
影响因子：
3.7
作者：
Rayner, Keith;Reichle, Erik D.;Pollatsek, Alexander
通讯作者：
Pollatsek, Alexander