REG Challenge 2008: A Shared Task Evaluation Event for Referring Expression Generation

自然语言生成（NLG）是自然语言处理（NLP）的一个子领域，主要致力于开发自动生成语言的计算方法，主要目的是节省文本制作过程（例如制作手册或信件的草稿），并改善对非语言信息的访问（例如为视障用户创建口头描述）。比较替代计算方法如何执行相同的任务（或“比较评估”）是巩固研究工作和技术进步的重要组成部分。一段时间以来，在许多NLP领域，比较评估倡议与相关的竞赛和活动已经很常见，它们被视为激励研究社区，创造有价值的新资源，并导致快速的技术进步。NLG有很强的评估传统，特别是在应用系统的用户评估方面，而且还用于NLG组件相对于非NLG基线或相同组件的不同版本的嵌入式评估。然而，很大程度上缺少的是可比较但独立开发的NLP系统和工具的比较评估结果。目前，只有两组这样的结果。在过去的两年里，NLG研究人员对比较评估越来越感兴趣。我们相信，比较评估举措将对NLG产生许多有益的影响，包括创造资源，将研究工作集中在特定任务上，以及吸引新的研究人员进入该领域。今年，我们组织了生成引用表达式的属性选择（ASGRE）挑战赛，这是一个试点NLG共享任务评估活动。参与率很高，NLG研究人员的反应也很热烈。因此，我们计划在2008年开展一项全面的NLG评估活动，即“指称表达生成（REG）挑战赛”。与由美国政府机构资助和指导的机器翻译和文档摘要这两个相邻领域的领先评估计划不同，ASGRE和REG挑战是社区主导的，基于英国的评估计划。该提案要求为2008年REG挑战中的数据编制和评估活动提供资金，使我们能够扩大共同任务和评估方案的范围，并使这一倡议保持以社区为基础和由联合王国领导。

Natural Language Generation (NLG) is the subfield of Natural Language Processing (NLP) that is concerned with developing computational methods for automatically generating language, with the primary aims of economising text-production processes (for example producing drafts of manuals or letters), and improving access to non-verbal information (for example creating verbal descriptions for visually impaired users). Comparing how well alternative computational methods perform the same task (or 'comparative evaluation') is an important component of the consolidation of research effort and technological progress in general. Comparative evaluation initiatives with associated competitions and events have been common in many NLP fields for some time, where they have been seen to galvanise research communities, create valuable new resources, and lead to rapid technological progress.NLG has strong evaluation traditions, in particular in user evaluations of application systems, but also in embedded evaluation of NLG components against non-NLG baselines or different versions of the same component. However, what has largely been missing are comparative evaluation results for comparable but independentlydeveloped NLP systems and tools. Right now, there are only two sets of such results. Over the past two years, NLG researchers have become increasingly interested in comparative evaluation. We believe that comparative evaluation initiatives will have many beneficial effects for NLG, including creation of resources, focussing research effort on specific tasks and attracting new researchers to the field. This year, we organised the Attribute Selection for Generating Referring Expressions (ASGRE) Challenge, which was a pilot NLG shared-task evaluation event. Participation was high and reactions from NLG researchers have been enthusiastic. We are therefore planning a full-scale NLG evalution initiative, the Referring Expressions Generation (REG) Challenge, for 2008. Unlike the two leading evaluation intitiatives in the neighbouring fields of Machine Translation and Document Summarisation, which are funded and directed by US government agencies, the ASGRE and REG Challenges are community-led, UK-based evaluation initiatives. This proposal requests funding for data preparation and evaluation activities in the 2008 REG Challenge, to enable us to extend the range of shared tasks and the evaluation programme, and to keep this initiative community-based and UK-led.