CiViL: Common-sense- and Visually-enhanced natural Language generation

人工智能中最引人注目的问题之一是创建能够在现实环境中使用自然语言进行交互的计算代理。像机器人这样的计算代理可以为社会提供多种好处，例如，它们可以用来照顾老龄化人口，充当同伴，可以用于技能培训，甚至可以在公共场所提供帮助。由于其复杂的跨学科性质，这些任务极具挑战性，涉及多个领域，包括自然语言生成，工程，计算机视觉和机器人技术。语言交流是最重要、最自然的互动方式。人类能够使用自然语言，利用常识知识，并根据之前与他人的互动来推断他人的背景，从而有效地相互交流。同时，即使遇到未知的实体和物体，它们也能成功地描述周围的环境。几十年来，研究人员一直试图通过自然语言重建人类交流的方式，尽管近年来取得了重大突破（如苹果的Siri或亚马逊的Alexa），但自然语言生成系统仍然缺乏推理、利用常识知识以及利用来自各种来源（如知识库、图像和视频）的多模态信息的能力。该项目旨在开发一个框架，用于常识和视觉增强的自然语言生成，使人类和人工代理（如机器人）之间的自然实时通信成为可能，从而实现人类和机器人之间的有效协作。由于动态环境的不确定性和交互方式的不确定性，人机交互对自然语言生成提出了新的挑战。例如，当机器人移动时，所处位置的机器人的视角会发生变化，因此它对世界的表征也会发生变化，这将导致当前最先进的方法失败，因为它们无法适应不断变化的环境。该项目旨在研究将各种模式联系起来的方法，同时考虑到它们的动态性质。为了实现自然、高效和直观的沟通能力，智能体还需要获得类似人类的综合知识和表达能力。外部知识库（如Wikipedia）在什么条件下可以被用来增强自然语言的生成，以及现有的知识库是否对语言的生成有用，都还需要探索。集成多模态数据进行语言生成的新方法将导致更健壮和高效的交互，并将对自然语言生成、社交机器人、计算机视觉和相关领域产生影响。反过来，这可能会产生全新的应用，例如解释电子健康治疗的确切程序，并增强以教育为目的的辅导系统。

One of the most compelling problems in Artificial Intelligence is to create computational agents capable of interacting in real-world environments using natural language. Computational agents such as robots can offer multiple benefits to society, for instance, they can be used to look after the ageing population, act as companions, can be used for skills training or even provide assistance in public spaces. These are extremely challenging tasks due to their complex interdisciplinary nature, which spans across several fields including Natural Language Generation, engineering, computer vision, and robotics. Communication through language is the most vital and natural way of interaction. Humans are able to effectively communicate with each other using natural language, utilising common-sense knowledge and by making inferences about other people's backgrounds based on previous interactions with them. At the same time, they can successfully describe their surroundings, even when encountering unknown entities and object. For decades, researchers have tried to recreate the way humans communicate through natural language and although there are major breakthroughs during recent years (such as Apple's Siri or Amazon's Alexa), Natural Language Generation systems still lack the ability to reason, exploit common-sense knowledge, and utilise multi-modal information from a variety of sources such as knowledge bases, images, and videos. This project aims to develop a framework for common-sense- and visually- enhanced Natural Language Generation that can enable natural real-time communication between humans and artificial agents such as robots to enable effective collaboration between humans and robots. Human-Robot Interaction poses additional challenges to Natural Language Generation due to uncertainty derived from the dynamic environments and the non-deterministic fashion of interaction. For instance, the viewpoint of a situated robot will change when the robot moves and hence its representation of the world, which will result in failure of current state-of-art methods, which are not able to adapt to changing environments. The project aims to investigate methods for linking various modalities, taking into account their dynamic nature. To achieve natural, efficient and intuitive communication capabilities, agents will also need to acquire human-like abilities in synthesising knowledge and expression. The conditions under which external knowledge bases (such as Wikipedia) can be used to enhance natural language generation still have to be explored as well as whether existing knowledge bases are useful for language generation. The novel ways to integrate multi-modal data for language generation will lead to more robust and efficient interactions and will have an impact on natural language generation, social robotics, computer vision, and related fields. This might, in turn, spawn entirely novel applications, such as explaining exact procedures for e-health treatments and enhance tutoring systems for educational purposes.