Visual Question Answering focused on properties (materials, shapes) and relations of small objects

研究的重点是学习物体的各种属性（如：表面类型，重量，大小，形状等），使用不同的方式提供的信息，即视觉和文本。在不同语义层次上对广泛的属性进行建模的主要应用是机器人操作。机器人系统需要物体的形状、体积、材料、质量、摩擦力等详细信息来进行正确的感知和抓取。本研究的主要目的与视觉问答任务密切相关，即为与给定图像相关的自然语言问题提供最准确的答案。另一个研究目标是提供从不同模态推断的对象属性的一致和结构化表示，并预测来自同一类别的未知对象实例的属性。该研究还将集中于为VQA提供一个新的实验框架，重点关注属性和关系，建立一种方法来比较用于识别物理属性的不同系统。本研究探索的方法将主要基于深度神经网络，它通常用于分类和识别任务。对于视觉数据，将探索神经网络图像分割以及从不同视图推断物体属性的新方法。文本数据也将使用自然语言处理中最先进的递归神经网络进行处理。这两种方法都将提取对象（视觉和文本）的特征，并提出将这些表征融合成一致的场景表征的方法。为了从这种表示中提取信息，我们将探讨一种符号程序执行的思想。符号程序是允许过滤场景表示或比较两个对象的属性的小模块。使用深度神经网络创建这样的程序，并提取被查询对象的属性。该研究将开发从现有的带有图片和插图的自然语言描述中推断物体相关属性的方法。开发的方法将在同意当前VQA评估协议的情况下提供，以便将其与现有方法进行比较。这项研究旨在提供一种新的、最先进的方法来解开由神经网络获得的视觉和文本信息的纠缠。此外，还将制定新的任务和基准，明确重点分析方法推断项目属性的能力。最后，该方法将通过为机器人提供有关其周围环境的丰富而精确的信息来增强机器人的自主能力。研究领域：-人工智能技术-图像和视觉计算-自然语言处理

The research is focused on learning various properties of objects (such as: surface type, weight, size, shape, etc.) using different modalities of information provided, i.e. visual and text. The main application of modelling a broad range of properties at different semantic levels is robotic manipulation. Robotic systems require detailed information about object's shape, volume, material, mass, friction, etc. to perform correct perception and grasping. The main objective of this research is closely related to the task of Visual Question Answering, which is to provide the most accurate answer for a natural language question related to a given image. Another re- search objective is to provide a consistent and structured representation of the object properties inferred from different modalities and to predict properties of unobserved instances of objects from the same category as the known ones. The research will also concentrate on providing a new experimental framework for VQA focused on properties and relations, establishing a way to compare different systems used for recognition of physical properties.The approach explored in this research will be primarily based on deep neural networks, which are commonly used in classification and recognition tasks. For the visual data, image segmentation with neural networks will be explored along with novel methods for inferring object properties from different views. Text data will also be processed with the use of recurrent neural networks being the state of the art in natural language processing. Both methods will extract features of objects (visual and textual) and the approach for fusing the representations into a consistent scene representation will be proposed. In order to extract information from such representation a symbolic program execution idea will be explored. Symbolic programs are small modules allowing to filter the scene representation or compare properties of two objects. Using deep neural networks such a program will be created, and the queried property of the object will be extracted. The research will develop methods to infer relevant properties of objects from existing natural language descriptions accompanied with pictures and illustrations. The developed approach will be provided in consent with current VQA evaluation protocols in order to compare it to the existing methods.The research is aiming to provide a new, state of the art approach for disentangling visual and textual information obtained by neural networks. In addition, a new task and benchmark will be formulated with explicit focus on analysis the capability of method to infer items' properties. Finally, the approach will be used to increase the autonomic capabilities of robots by providing them with rich and precise information about their surroundings.EPSRC research areas:- Artificial intelligence technologies- Image and vision computing- Natural language processing