Methods for Evaluating the Fluency of Automatically Simplified Texts with Deaf and Hard-of-Hearing Adults at Various Literacy Levels
Methods for Evaluating the Fluency of Automatically Simplified Texts with Deaf and Hard-of-Hearing Adults at Various Literacy Levels
复制标题
DOI：
10.1145/3491102.3517566
复制
发表时间：
2022-04
期刊：
影响因子：
--
通讯作者：
Oliver Alonzo;J. Trussell;Matthew Watkins;Sooyeon Lee;Matt Huenerfauth
中科院分区：
文献类型：
--
作者：
Oliver Alonzo;J. Trussell;Matthew Watkins;Sooyeon Lee;Matt Huenerfauth
研究表明，聋人和听力障碍(DHH)成年人对由自动文本简化(ATS)支持的阅读辅助工具有好处和兴趣，ATS是一项技术，其开发得益于特定用户群体的评估。虽然以前的工作为评估DHH成人的文本复杂性提供了指导，但研究人员缺乏评估自动简化文本的流畅性的指导，这可能包含简化过程中的错误。因此，我们在不同识字水平的DHH参与者之间进行了方法论研究，以评估用于评估受控于不同流利程度的文本的指标(包括阅读速度；理解问题；以及对可理解性、可读性、语法和系统性能的主观判断)的有效性。阅读速度和语法判断有效地区分了不同识字水平的参与者的流利程度。然而，可读性和可理解性判断只在识字水平较高的参与者中起作用。我们的研究结果为设计DHH参与者的ATS评估提供了方法学指导。
Research has revealed benefits and interest among Deaf and Hard-of-Hearing (DHH) adults in reading-assistance tools powered by Automatic Text Simplification (ATS), a technology whose development benefits from evaluations by specific user groups. While prior work has provided guidance for evaluating text complexity among DHH adults, researchers lack guidance for evaluating the fluency of automatically simplified texts, which may contain errors from the simplification process. Thus, we conduct methodological research on the effectiveness of metrics (including reading speed; comprehension questions; and subjective judgements of understandability, readability, grammaticality, and system performance) for evaluating texts controlled to be at different levels of fluency, when measured among DHH participants at different literacy levels. Reading speed and grammaticality judgements effectively distinguished fluency levels among participants across literacy levels. Readability and understandability judgements, however, only worked among participants with higher literacy. Our findings provide methodological guidance for designing ATS evaluations with DHH participants.