Grounding Statistical Machine Translation in Perception and Action

我们提出了一个基于统计机器翻译（SMT）在感知和行动方面的研究方案。机器翻译中意义保存的关键概念将在世界状态的情景语境中进行定义和评估。优化和评价的标准不再是孤立句的翻译充分性，不再是忽视语篇语境，不再是忽视译文交际的外在任务。相反，基于基础的SMT将侧重于在具体任务中对人类感知和行动的适当意义转移。例如，一个成功的自然语言指令的翻译可以使一个动作遵循该指令。在游戏规则说明的上下文中，这意味着游戏规则的描述被成功翻译，如果正确的游戏移动可以仅基于翻译执行。在翻译视觉场景描述的情况下，要传递的意义可以直接与图像联系起来：如果单语使用者能够仅根据翻译结果在许多相似图像中识别出相应的图像，那么对视觉场景的描述就可以成功翻译。基础SMT引入了针对特定任务的翻译质量评估的概念。它还提供了将特定于任务的翻译反馈部署为学习SMT系统的数据的机会。这可以在线完成，例如，通过从反馈中进行机器学习，也可以离线完成，例如，通过将用户创建和用户纠正的翻译存储为并行数据。我们项目的主要挑战如下：+提供和推广翻译数据创建的新机制。我们提出了一个游戏化框架，可以通过模拟和人类游戏来创建大量的后期编辑翻译，用于训练和评估SMT系统。+设计一个机器学习框架，能够在接地场景中实现自动学习。我们将重点关注基于反应的学习，在这种学习中，学习者唯一可用的监督信号是来自世界上行动的反应。+在各种具体任务中对接地SMT进行实验，反馈学习和评价。我们将专注于游戏场景，其中反馈是基于计算机或人类对机器翻译的响应而执行的动作。

We propose a research program for grounding statistical machine translation (SMT) in perception and action. The crucial concept of preservation of meaning in machine translation will be defined and evaluated in the situational context of a world state. The criteria for optimization and evaluation are no longer translational adequacy of isolated sentences, ignoring the context of the discourse and ignoring the extrinsic task in which the translation is communicated. Instead, grounded SMT will focus on adequate meaning transfer with respect to human perception and action in a concrete task. For example, a successful translation of a natural language instruction enables an action that follows the instruction. In the context of instructions on game rules, this means that a description of a game rule is translated successfully, if correct game moves can be performed based only on the translation. In case of translating a description of a visual scene, the meaning to be transferred can be directly tied to the image: A description of a visual scene is translated successfully, if a monolingual speaker can identify the corresponding image among many similar images, based only on the translation. Grounded SMT introduces the concept of a task-specific evaluation of translation quality. It also offers the opportunity to deploy task-specific feedback on translations as data for learning SMT systems. This can be done online, e.g., by machine learning from feedback, or offline, e.g., by storing user-created and user-corrected translations as parallel data. The main challenges of our project will be as follows:+ Provide and popularize new mechanisms for translation data creation. We propose a gamification framework that potentially enables the creation of very large amounts of post-edited translations for training and evaluating SMT systems via simulated and human gameplay.+ Devise a machine learning framework that enables automatic learning in grounded scenarios. We will focus on response-based learning in which the only supervision signal available to the learner is the response from acting in the world. + Conduct experiments on grounding SMT in various concrete tasks, in which feedback for learning and evaluation will be provided. We will focus on game scenarios in which feedback is based on actions executed by a computer or a human in response to machine translations.