Grounding Statistical Machine Translation in Perception and Action

我们提出了一个基于感知和行为的基础统计机器翻译的研究方案。机器翻译中的意义保留这一关键概念将在一个世界国家的情景背景下定义和评估。优化和评价的标准不再是孤立的句子的翻译充分性，忽视了语篇的语境和翻译交流的外在任务。相反，扎根的SMT将专注于关于人类在具体任务中的感知和行动的适当的意义传递。例如，自然语言指令的成功翻译使遵循该指令的操作成为可能。在关于游戏规则的指令的上下文中，这意味着如果仅基于转换可以执行正确的游戏移动，则游戏规则的描述被成功地转换。在翻译视觉场景的描述的情况下，要传递的含义可以直接绑定到图像：如果单语说话者能够仅基于翻译在许多相似图像中识别对应的图像，则视觉场景的描述被成功地翻译。扎根的SMT引入了特定于任务的翻译质量评估的概念。它还提供了部署特定于任务的翻译反馈作为学习SMT系统的数据的机会。这可以在线地完成，例如通过来自反馈的机器学习，或者离线，例如通过将用户创建和用户校正的翻译存储为并行数据。我们项目的主要挑战如下：+提供并推广翻译数据创建的新机制。我们提出了一个游戏化框架，它潜在地能够创建非常大量的编辑后翻译，用于通过模拟和人类游戏来培训和评估SMT系统。+设计一个机器学习框架，使其能够在接地场景中自动学习。我们将专注于基于反应的学习，在这种学习中，学习者可用的唯一监督信号是来自于在世界上行动的反应。+在各种具体任务中进行SMT接地实验，并提供学习和评估反馈。我们将专注于游戏场景，其中反馈基于计算机或人类对机器翻译的响应执行的操作。

We propose a research program for grounding statistical machine translation (SMT) in perception and action. The crucial concept of preservation of meaning in machine translation will be defined and evaluated in the situational context of a world state. The criteria for optimization and evaluation are no longer translational adequacy of isolated sentences, ignoring the context of the discourse and ignoring the extrinsic task in which the translation is communicated. Instead, grounded SMT will focus on adequate meaning transfer with respect to human perception and action in a concrete task. For example, a successful translation of a natural language instruction enables an action that follows the instruction. In the context of instructions on game rules, this means that a description of a game rule is translated successfully, if correct game moves can be performed based only on the translation. In case of translating a description of a visual scene, the meaning to be transferred can be directly tied to the image: A description of a visual scene is translated successfully, if a monolingual speaker can identify the corresponding image among many similar images, based only on the translation. Grounded SMT introduces the concept of a task-specific evaluation of translation quality. It also offers the opportunity to deploy task-specific feedback on translations as data for learning SMT systems. This can be done online, e.g., by machine learning from feedback, or offline, e.g., by storing user-created and user-corrected translations as parallel data. The main challenges of our project will be as follows:+ Provide and popularize new mechanisms for translation data creation. We propose a gamification framework that potentially enables the creation of very large amounts of post-edited translations for training and evaluating SMT systems via simulated and human gameplay.+ Devise a machine learning framework that enables automatic learning in grounded scenarios. We will focus on response-based learning in which the only supervision signal available to the learner is the response from acting in the world. + Conduct experiments on grounding SMT in various concrete tasks, in which feedback for learning and evaluation will be provided. We will focus on game scenarios in which feedback is based on actions executed by a computer or a human in response to machine translations.