Visual Question Answering focused on properties (materials, shapes) and relations of small objects

研究重点是学习物体的各种属性（如：表面类型，重量，大小，形状等）。使用提供信息的不同方式，即视觉和文字。在不同语义层次上对广泛的属性进行建模的主要应用是机器人操作。机器人系统需要关于物体的形状、体积、材料、质量、摩擦力等的详细信息来执行正确的感知和抓取。本研究的主要目标是密切相关的视觉问题的提问，这是提供最准确的答案与给定的图像相关的自然语言问题的任务。另一个研究目标是提供从不同模态推断的对象属性的一致和结构化表示，并预测来自与已知对象相同类别的对象的未观察实例的属性。该研究还将致力于为VQA提供一个新的实验框架，重点是属性和关系，建立一种方法来比较用于识别物理属性的不同系统。本研究中探索的方法将主要基于深度神经网络，这是常用的分类和识别任务。对于视觉数据，将沿着探索用神经网络进行图像分割，以及从不同视角推断物体属性的新方法。还将使用自然语言处理中最先进的递归神经网络来处理文本数据。这两种方法都将提取对象的特征（视觉和文本），并将提出将表示融合为一致的场景表示的方法。为了从这种表示中提取信息，将探索符号程序执行的想法。符号程序是允许过滤场景表示或比较两个对象的属性的小模块。使用深度神经网络将创建这样的程序，并提取对象的查询属性。该研究将开发从现有的自然语言描述中推断对象相关属性的方法，并伴有图片和插图。开发的方法将提供同意与当前的VQA评估协议，以比较它与现有的methods.The研究的目的是提供一个新的，最先进的方法解开视觉和文本信息的神经网络。此外，一个新的任务和基准将制定明确的重点是分析的能力的方法推断项目的属性。最后，该方法将用于通过向机器人提供关于其周围环境的丰富而精确的信息来提高机器人的自主能力。EPSRC研究领域：-人工智能技术-图像和视觉计算-自然语言处理

The research is focused on learning various properties of objects (such as: surface type, weight, size, shape, etc.) using different modalities of information provided, i.e. visual and text. The main application of modelling a broad range of properties at different semantic levels is robotic manipulation. Robotic systems require detailed information about object's shape, volume, material, mass, friction, etc. to perform correct perception and grasping. The main objective of this research is closely related to the task of Visual Question Answering, which is to provide the most accurate answer for a natural language question related to a given image. Another re- search objective is to provide a consistent and structured representation of the object properties inferred from different modalities and to predict properties of unobserved instances of objects from the same category as the known ones. The research will also concentrate on providing a new experimental framework for VQA focused on properties and relations, establishing a way to compare different systems used for recognition of physical properties.The approach explored in this research will be primarily based on deep neural networks, which are commonly used in classification and recognition tasks. For the visual data, image segmentation with neural networks will be explored along with novel methods for inferring object properties from different views. Text data will also be processed with the use of recurrent neural networks being the state of the art in natural language processing. Both methods will extract features of objects (visual and textual) and the approach for fusing the representations into a consistent scene representation will be proposed. In order to extract information from such representation a symbolic program execution idea will be explored. Symbolic programs are small modules allowing to filter the scene representation or compare properties of two objects. Using deep neural networks such a program will be created, and the queried property of the object will be extracted. The research will develop methods to infer relevant properties of objects from existing natural language descriptions accompanied with pictures and illustrations. The developed approach will be provided in consent with current VQA evaluation protocols in order to compare it to the existing methods.The research is aiming to provide a new, state of the art approach for disentangling visual and textual information obtained by neural networks. In addition, a new task and benchmark will be formulated with explicit focus on analysis the capability of method to infer items' properties. Finally, the approach will be used to increase the autonomic capabilities of robots by providing them with rich and precise information about their surroundings.EPSRC research areas:- Artificial intelligence technologies- Image and vision computing- Natural language processing