Developing Foundation Model Capabilities for Video Understanding in the Open World

本项目的目标是开发用于视频理解的开放世界深度学习模型，允许用户使用自然语言描述来询问有关视频内容的问题。开发的方法将利用预先训练的基础模型，而不是从头开始训练深度学习模型。基础模型是一种基于大量数据训练的机器学习模型，这些数据可以适用于解决各种下游任务。扩展基础模型来解决特定问题通常需要比从头开始训练更少的数据，并提高专家模型的泛化能力。将开发各种方法来解决开放世界的图像级问题，这些问题需要通过预先训练的基础模型进行自然语言输入。来自这些发展的见解将启发构建解决视频中类似问题的模型。问题的例子包括计算图像中的文本指定对象和视频中的重复，以及回答关于场景中对象的面积、形状和结构的询问。开发的模型将允许用户通过在推理时提供关于感兴趣的类的文本输入来解决任意类的问题，而不是针对特定类解决问题。重要的是，使这种开放世界模型适应新的班级不需要额外的培训或数据，即使在培训期间看不到班级。因此，这项工作将导致人工智能系统更容易为普通公众所访问，他们可能无法访问训练特定类别的模型通常所需的大量标记数据和计算。开发图像理解模型，允许用户使用自然语言就图像内容提问。利用第(1)步中的见解和方法开发具有类似视频理解功能的模型，允许用户使用自然语言提出有关视频内容的问题。例如，在步骤(1)中开发的使用文本对图像中的对象进行计数的模型可以激励在步骤(2)中使用文本对视频中的对象进行计数的模型。重复第(1)步和第(2)步，添加更多功能。研究方法的新颖性虽然在图像检索、对象检测和实例分割等任务中利用预先训练的视觉语言基础模型进行了大量探索，但在视频中类似的开发探索较少。这是因为从视频中学习由于额外的时间维度而更加复杂。此外，在这个项目中开发的方法将包括新的深度学习架构，这些架构更通用，在现有任务中表现更好，或者解决新的问题，例如使用自然语言描述视频中的重复计数，以及回答关于对象的大小、形状和结构的任意自然语言查询。与EPSRC的战略和研究区域保持一致这个项目涉及“人工智能技术”研究领域。是否有公司或合作者参与？没有。

DescriptionThe goal of this project is to develop open-world deep learning models for video understanding that allow users to ask queries about video content using natural language descriptions. Rather than training deep learning models from scratch, the methods developed will leverage pre-trained foundation models. A foundation model is a machine learning model trained on large quantities of data that can be adapted to solve a wide variety of downstream tasks. Extending a foundation model to solve a specific problem usually requires less data than training it from scratch and improves the generalizability of the specialist model. Methods will be developed to solve open-world image-level problems requiring natural language input with pre-trained foundation models. Insights from these developments will inspire the construction of models that solve analogous problems in videos. Examples of problems include counting text-specified objects in images and repetitions in videos and answering queries about the area, shape, and structure of objects in a scene. Rather than solving a problem for a particular class, the models developed will allow users to solve the problem for any arbitrary class by providing text input about the class of interest at inference time. Importantly, adapting such open-world models to new classes would require no additional training or data, even if the class were unseen during training. Hence, this work will result in AI systems that are more accessible to the general public, who may not have access to the large quantities of labelled data and compute typically necessary to train class-specific models.Aims & Objectives1. Develop models for image understanding that allow users to ask questions on the image content using natural language.2. Leverage insights and methods from step (1) to develop models with similar capabilities for video understanding that allow users to ask questions on video content using natural language. For instance, a model developed to count objects in images using text in step (1) could inspire a model to count objects in videos using text in step (2).3. Iterate on steps (1) and (2), adding more capabilities. Novelty of the Research MethodologyWhile leveraging pre-trained vision-language foundation models for tasks such as image retrieval, object detection, and instance segmentation has been significantly explored for images, similar developments have been less explored for videos. This is because learning from videos is more complex due to an additional temporal dimension. Furthermore, methods developed in this project will include novel deep learning architectures that are more general and perform better at existing tasks or that solve new problems such as repetition counting in videos using natural language descriptions and answering arbitrary natural language queries about the size, shape, and structure of objects.Alignment to the EPSRC's Strategies & Research AreasThis project relates to the "Artificial Intelligence Technologies" research area.Any Companies or Collaborators Involved?No.