CiViL: Common-sense- and Visually-enhanced natural Language generation

人工智能中最引人注目的问题之一是创建能够在现实环境中使用自然语言进行交互的计算代理。机器人等计算代理可以为社会带来多方面的好处，例如，它们可以用来照顾老龄化的人口，充当伴侣，可以用于技能培训，甚至可以在公共场所提供帮助。由于它们复杂的跨学科性质，这些任务都是极具挑战性的任务，涉及自然语言生成、工程、计算机视觉和机器人学等多个领域。通过语言进行交流是最重要、最自然的互动方式。人类能够利用自然语言，利用常识知识，并根据之前与他人的互动来推断他人的背景，从而有效地相互交流。同时，他们可以成功地描述他们的环境，即使在遇到未知的实体和对象时也是如此。几十年来，研究人员一直试图重建人类通过自然语言进行交流的方式，尽管近年来取得了重大突破(如苹果的Siri或亚马逊的Alexa)，但自然语言生成系统仍然缺乏推理能力，开发常识知识，并利用来自各种来源的多模式信息，如知识库、图像和视频。该项目旨在开发一个常识和视觉增强的自然语言生成框架，使人类和机器人等人工代理之间能够进行自然的实时交流，从而实现人类和机器人之间的有效协作。由于动态环境和非确定性交互方式的不确定性，人-机器人交互给自然语言生成带来了额外的挑战。例如，当机器人移动时，机器人的视点会改变，从而改变它对世界的表示，这将导致当前最先进的方法失败，这些方法不能适应不断变化的环境。该项目旨在调查将各种模式联系起来的方法，同时考虑到它们的动态性质。为了实现自然、高效和直观的通信能力，代理还需要获得类似人类的合成知识和表达的能力。在何种条件下，外部知识库(如维基百科)可以用来促进自然语言的生成，以及现有的知识库是否对语言生成有用，仍有待探索。集成多模式数据用于语言生成的新方法将导致更健壮和高效的交互，并将对自然语言生成、社会机器人、计算机视觉和相关领域产生影响。反过来，这可能会催生全新的应用程序，例如解释电子健康治疗的确切程序，并增强用于教育目的的辅导系统。

One of the most compelling problems in Artificial Intelligence is to create computational agents capable of interacting in real-world environments using natural language. Computational agents such as robots can offer multiple benefits to society, for instance, they can be used to look after the ageing population, act as companions, can be used for skills training or even provide assistance in public spaces. These are extremely challenging tasks due to their complex interdisciplinary nature, which spans across several fields including Natural Language Generation, engineering, computer vision, and robotics. Communication through language is the most vital and natural way of interaction. Humans are able to effectively communicate with each other using natural language, utilising common-sense knowledge and by making inferences about other people's backgrounds based on previous interactions with them. At the same time, they can successfully describe their surroundings, even when encountering unknown entities and object. For decades, researchers have tried to recreate the way humans communicate through natural language and although there are major breakthroughs during recent years (such as Apple's Siri or Amazon's Alexa), Natural Language Generation systems still lack the ability to reason, exploit common-sense knowledge, and utilise multi-modal information from a variety of sources such as knowledge bases, images, and videos. This project aims to develop a framework for common-sense- and visually- enhanced Natural Language Generation that can enable natural real-time communication between humans and artificial agents such as robots to enable effective collaboration between humans and robots. Human-Robot Interaction poses additional challenges to Natural Language Generation due to uncertainty derived from the dynamic environments and the non-deterministic fashion of interaction. For instance, the viewpoint of a situated robot will change when the robot moves and hence its representation of the world, which will result in failure of current state-of-art methods, which are not able to adapt to changing environments. The project aims to investigate methods for linking various modalities, taking into account their dynamic nature. To achieve natural, efficient and intuitive communication capabilities, agents will also need to acquire human-like abilities in synthesising knowledge and expression. The conditions under which external knowledge bases (such as Wikipedia) can be used to enhance natural language generation still have to be explored as well as whether existing knowledge bases are useful for language generation. The novel ways to integrate multi-modal data for language generation will lead to more robust and efficient interactions and will have an impact on natural language generation, social robotics, computer vision, and related fields. This might, in turn, spawn entirely novel applications, such as explaining exact procedures for e-health treatments and enhance tutoring systems for educational purposes.