Analysis and automatic estimation of the quality of machine-generated texts

本研究项目涉及机器生成文本的体验质量。项目的重点在于两种文本类型：机器翻译和自动摘要。该项目旨在识别感知质量维度，提供主观方法来量化这些质量维度，自动确定与文本质量相关的可提取参数，从而开发预测模型，可以提供对机器生成文本整体质量的估计。预测模型是在各种德语和英语语料库的帮助下开发的。基于这些语料库，正在与研究参与者（通过众包和在实验室）进行测试，定义和验证与质量估计相关的维度。对于每个确定的质量维度，相关的文本参数被自动地从文本中提取出来，然后与维度的主观评价相关联。基于参数的预测模型正在为两种文本类型- -为单独的质量维度以及为整体质量- -构建，并就文本类型之间的差异进行分析。最后，在一个独立的测试序列中对预测模型的泛化性进行了测试。

This research project deals with the Quality of Experience of machine-generated texts. The focus within the project lies on the two text types machine translation and automatic summarization. The project aims to identify perceptual quality dimensions, provide subjective methods to quantify these quality dimensions, automatically determine extractable parameters which correlate with the text quality, and thereby develop predictive models which can provide an estimation of the overall quality of a machine-generated text. The predictive models are being developed with the help of various corpora for German and English. Based on these corpora, dimensions that are relevant for the quality estimation are being defined and verified in tests with study participants (through crowdsourcing and in the laboratory). For each identified quality dimension, relevant text parameters are automatically being extracted from the texts which are then correlated with the subjective evaluations of the dimension. Predictive models based on the parameters are being constructed for both text types – for separate quality dimensions as well as for the overall quality – and analyzed concerning the differences between the text types. Finally, the predictive models are being tested regarding their generalizability in an independent test series.