Generation Challenges 2011: Towards a Surface Realisation Shared Task

计算机现在可以很好地完成一些写作任务（例如抄写和拼写检查），但我们仍然缺乏好的计算解决方案来完成涉及创建新文本的写作任务（而不是键入或检查现有文本）。自然语言生成（NLG）是计算机科学的一个分支，旨在通过开发用于口语和书面语的计算生成的方法和工具来解决这一缺陷。NLG技术具有广泛的潜在应用，包括提高文本生产过程的效率（例如自动信函和报告撰写），以及以口头形式提供信息，否则这些信息将无法获得（例如盲人）或更耗时的处理（例如将天气数据转换为文本摘要）。然而，NLG才刚刚开始发挥这一潜力。其中一个原因是，NLG直到最近才采用比较形式的评估，而这对于有效比较替代方法、整合和集体科学进步至关重要。NLG领域的评价传统是对完整系统进行面向用户和基于任务的评价。这种传统与在自然语言处理（NLP）的其他领域占主导地位的比较评估范式非常不同，在NLP中，共享数据资源、内在的、自动计算的度量和人为的质量评级提供了时间效率和低成本的方法来比较新系统和技术与现有方法。相比之下，在NLG，直到几年前，根本没有对独立开发的替代方法进行比较评价。然而，如果没有比较评价，一个研究领域就不可能有巩固或集体进步，个别研究人员和小组或多或少只能各自独立地取得进展。GenChal倡议已经牢固地建立了NLG的比较评估，并产生了数据集和软件工具来支持它。过去的共享任务已经解决了参考生成和特定应用的子领域，现在是时候解决一个更雄心勃勃的挑战了。表面实现任务构成了本提案的核心，我们的目标是真正具有突破性的东西，在实际应用中具有巨大的潜在用途：开发新一代表面实现剂，可以直接比较，并且由于它们来自共同的输入，可以相互替代。最终，这将意味着数据到文本的生成、机器翻译、摘要和对话系统（在各个领域之间）将直接受益于一系列可重用的实现组件的可用性，系统构建者可以对这些组件进行测试，以确定哪一个最适合他们的目的，这在以前是不可能的。

Computers can now perform some writing tasks well (e.g. transcription and spell checking), but we still lack goodcomputational solutions for writing tasks which involve the creation of new text (as opposed to the typing orchecking of existing text). Natural language generation (NLG) is the branch of computer science that aims toaddress this lack, by developing methods and tools for the computational generation of spoken and writtenlanguage. NLG technology has a vast range of potential applications, including increasing the efficiency oftext-production processes (e.g. automated letter and report writing) and making information available in verbal formthat would otherwise be inaccessible (e.g. to the blind) or more time-consuming to process (e.g. converting weatherdata to a textual summary). However, NLG is only just beginning to fulfil this potential. Among the reasons is thefact that NLG did not until recently employ comparative forms of evaluation, as are essential for effectivecomparison of alternative approaches, consolidation and collective scientific progress.The NLG field's evaluation tradition lies in user-oriented and task-based evaluation of complete systems.This tradition is very different from the comparative evaluation paradigms that are predominant in otherareas of Natural Language Processing (NLP) where shared data resources, intrinsic, automatically computedmetrics, and human ratings of quality provide time-efficient and low-cost ways of comparing new systems andtechniques against existing approaches. In contrast, in NLG, until a few years ago, there simply was no comparative evaluation of independently developed alternative approaches. Yet without comparative evaluation there can be noconsolidation or collective progress in a field of research, and individual researchers and groups are left toprogress more or less separately. The GenChal initiative has firmly established comparative evaluation in NLG andproduced data sets and software tools to support it. Past shared tasks have addressed the subfield of referencegeneration and specific applications, and the time is now right to tackle a more ambitious challenge.With the Surface Realisation Task that forms the core of the present proposal, we are aiming for something trulygroundbreaking and of great potential use in practical applications: the development of a new generation of surfacerealisers that can be directly compared and, because they work from common input, can be substituted for eachother. Ultimately, this will mean that data-to-text generation, MT, summarisation and dialogue systems (amongother fields) will directly benefit from the availability of a range of reusable realisation components which systembuilders can test to determine which is best for their purpose, something that has not been possible before.