RECOLAGE: Real-Time Vision-Grounded Collaborative Language Generation

当前的口语对话系统在产生口头输出时大多使用简单的预制话语。从理论上讲，数据驱动的自然语言生成（NLG）系统以灵活的方式将非语言数据（例如图像）映射到语言输出，有望为人类和机器之间更流畅和自然的交互提供手段。不幸的是，在国家的最先进的NLG系统的某些假设是大量定制的totext，并不能很容易地转移到口语互动：基本上，现有的框架设想的NLG作为一个自主的过程，是完全解耦的对话者和视觉动态环境。这种假设是特别有问题的口头任务为导向的互动在视觉环境。在这里，人类对话者期望说话者即使在说话时也能合作，对并发事件做出反应，并相应地调整他的话语，例如，如果需要的话，扩展或修改它们。RECOLAGE的中心目标是开发一个数据驱动的模型，用于在视觉接地对话系统中实时和协作的语言生成，该模型在用户的非语言行为和系统的语言行为之间保持密切的反馈回路。该框架将实现一种视觉和会话语言基础的方法，该方法能够实时打包其口头输出，并在世界的不确定性或变化使其变得可取时对其进行修改。产生这种说话行为需要协调和交错传统上顺序处理的任务，即预测系统动作（动作管理，AM）、话语生成（自然语言生成，NLG）和语音合成（语音合成，SYN）。RECOLAGE将AM建模为一个连续的决策过程，为NLG和SYN安排任务;这些反过来又保留了他们必须做出的语言决策（说哪些词，以及如何说）的自主权，但适应于在最小的组块上操作，并在强大的相互上下文约束下。在申请人之前大量相关工作的基础上，RECOLAGE将采用数据驱动的方法，通过机器学习技术优化语言决策。

Current spoken dialogue systems mostly use simple canned utterances when producing verbal output. Theoretically, data-driven natural language generation (NLG) systems, that map non-verbal data (e.g. images) to verbal output in a flexible way, promise to provide means for more fluid and natural interaction between humans and machines. Unfortunately, certain assumptions made in state-of-the-art NLG systems are heavily tailored totext and cannot easily be transferred to spoken interaction: essentially, existing frameworks conceive of NLG as an autonomous process that is entirely decoupled from an interlocutor and a visual dynamic environment. This assumption is particularly problematic for spoken task-oriented interaction in visual contexts. Here, human interlocutors expect the speaker to behave collaboratively even while he is speaking, to react to concurrent events, and to adapt his utterances accordingly, e.g. extending or revising them if needed. The central objective of RECOLAGE is to develop a data-driven model for real-time and collaborative language generation in visually grounded dialogue systems, that maintains a close feedback loopbetween a user’s non-verbal actions and the system’s verbal actions. This framework will implement an approach to visual and conversational language grounding that is able to package its verbal output in real-time, and revise it if uncertainty or changes in the world make this desirable.Producing speaking behaviour of this kind requires the coordination and interleaving of tasks that are traditionally handled sequentially, namely the prediction of system actions (action management, AM), the generation of utterances (natural language generation, NLG), and the synthesis of speech (speech synthesis, SYN). RECOLAGE will model AM as a continuous decision making process which schedules tasks for NLG and SYN; these in turn retain autonomy over the linguistic decisions they have to make (which words to say, and how to say them), but are adapted to operate on minimal chunks and under strong mutual contextual constraints. Building on the substantial relevant prior work of the applicants, RECOLAGE will follow a data-driven approach, where linguistic decisions will be optimised through machine learning techniques.