System Building Cost vs. Output Quality in Data-to-Text Generation
System Building Cost vs. Output Quality in Data-to-Text Generation
复制标题
DOI：
10.3115/1610195.1610198
复制
发表时间：
2009-03
期刊：
影响因子：
--
通讯作者：
A. Belz;Eric Kow
中科院分区：
文献类型：
--
作者：
A. Belz;Eric Kow
数据到文本生成系统往往是基于知识和手工构建的，这限制了它们的可重用性，并使它们的创建和维护耗费时间和成本。自动化（部分）系统构建过程的方法是存在的，但是这些方法是否冒着输出质量损失的风险？本文研究了发电系统建设中的成本/质量权衡问题。我们比较了四个主要由自动技术创建的新的数据到文本系统和六个主要由手工技术创建的相同领域的现有系统。我们使用内在自动度量和人类质量评级来评估这十个系统。我们发现增加系统构建自动化的程度并不一定会导致输出质量的降低。我们进一步发现，标准的自动评估度量低估了手工制作系统的质量，而高估了自动创建系统的质量。
Data-to-text generation systems tend to be knowledge-based and manually built, which limits their reusability and makes them time and cost-intensive to create and maintain. Methods for automating (part of) the system building process exist, but do such methods risk a loss in output quality? In this paper, we investigate the cost/quality trade-off in generation system building. We compare four new data-to-text systems which were created by predominantly automatic techniques against six existing systems for the same domain which were created by predominantly manual techniques. We evaluate the ten systems using intrinsic automatic metrics and human quality ratings. We find that increasing the degree to which system building is automated does not necessarily result in a reduction in output quality. We find furthermore that standard automatic evaluation metrics underestimate the quality of handcrafted systems and over-estimate the quality of automatically created systems.