Grounding Statistical Machine Translation in Perception and Action

我们提出了一个研究计划，为接地统计机器翻译（SMT）的感知和行动。在机器翻译中保存意义的关键概念将在世界国家的情境背景下定义和评估。优化和评估的标准不再是孤立句子的翻译适当性，忽略了语篇的上下文，忽略了翻译所传达的外在任务。相反，接地SMT将专注于在具体任务中与人类感知和行动有关的适当意义转移。例如，自然语言指令的成功翻译使得能够执行指令之后的动作。在关于游戏规则的指令的上下文中，这意味着游戏规则的描述被成功地翻译，如果仅基于翻译可以执行正确的游戏移动。在翻译视觉场景的描述的情况下，要传递的含义可以直接与图像联系在一起：如果单语说话者可以仅基于翻译在许多相似图像中识别对应的图像，则视觉场景的描述被成功翻译。Grounded SMT引入了针对特定任务的翻译质量评估概念。它还提供了将特定于任务的翻译反馈作为学习SMT系统的数据进行部署的机会。这可以在线完成，例如，通过从反馈的机器学习，或者离线，例如， 通过将用户创建的和用户校正的翻译存储为并行数据。我们的项目的主要挑战将如下：+提供和推广翻译数据创建的新机制。我们提出了一个游戏化框架，可以通过模拟和真人游戏创建大量的后期编辑翻译，用于训练和评估SMT系统。设计一个机器学习框架，在基础场景中实现自动学习。我们将专注于基于反应的学习，在这种学习中，学习者唯一可用的监督信号是在世界中行动的反应。+ 在各种具体任务中进行SMT接地实验，并提供学习和评估反馈。我们将专注于游戏场景，其中反馈是基于计算机或人类响应机器翻译而执行的动作。

We propose a research program for grounding statistical machine translation (SMT) in perception and action. The crucial concept of preservation of meaning in machine translation will be defined and evaluated in the situational context of a world state. The criteria for optimization and evaluation are no longer translational adequacy of isolated sentences, ignoring the context of the discourse and ignoring the extrinsic task in which the translation is communicated. Instead, grounded SMT will focus on adequate meaning transfer with respect to human perception and action in a concrete task. For example, a successful translation of a natural language instruction enables an action that follows the instruction. In the context of instructions on game rules, this means that a description of a game rule is translated successfully, if correct game moves can be performed based only on the translation. In case of translating a description of a visual scene, the meaning to be transferred can be directly tied to the image: A description of a visual scene is translated successfully, if a monolingual speaker can identify the corresponding image among many similar images, based only on the translation. Grounded SMT introduces the concept of a task-specific evaluation of translation quality. It also offers the opportunity to deploy task-specific feedback on translations as data for learning SMT systems. This can be done online, e.g., by machine learning from feedback, or offline, e.g., by storing user-created and user-corrected translations as parallel data. The main challenges of our project will be as follows:+ Provide and popularize new mechanisms for translation data creation. We propose a gamification framework that potentially enables the creation of very large amounts of post-edited translations for training and evaluating SMT systems via simulated and human gameplay.+ Devise a machine learning framework that enables automatic learning in grounded scenarios. We will focus on response-based learning in which the only supervision signal available to the learner is the response from acting in the world. + Conduct experiments on grounding SMT in various concrete tasks, in which feedback for learning and evaluation will be provided. We will focus on game scenarios in which feedback is based on actions executed by a computer or a human in response to machine translations.