Visual AI: An Open World Interpretable Visual Transformer

视觉人工智能：开放世界的可解释视觉转换器

• 批准号: EP/T028572/1
• 负责人: Andrew Zisserman
• 金额: $ 753.32万
• 依托单位: University of Oxford
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2020
• 资助国家: 英国
• 起止时间: 2020 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FT028572%2F1
• 关键词: Visual; AI; Open; World; Interpretable

With the advent of deep learning and the availability of big data, it is now possible to train machine learning algorithms for a multitude of visual tasks, such as tagging personal image collections in the cloud, recognizing faces, and 3D shape scanning with phones. However, each of these tasks currently requires training a neural network on a very large image dataset specifically collected and labelled for that task. The resulting networks are good experts for the target task, but they only understand the 'closed world' experienced during training and can 'say' nothing useful about other content, nor can they be applied to other tasks without retraining, nor do they have an ability to explain their decisions or to recognise their limitations. Furthermore, current visual algorithms are usually 'single modal', they 'close their ears' to the other modalities (audio, text) that may be readily available.The core objective of the Programme is to develop the next generation of audio-visual algorithms that does not have these limitations. We will carry out fundamental research to develop a Visual Transformer capable of visual analysis with the flexibility and interpretability of a human visual system, and aided by the other 'senses' - audio and text. It will be able to continually learn from raw data streams without requiring the traditional 'strong supervision' of a new dataset for each new task, and deliver and distill semantic and geometric information over a multitude of data types (for example, videos with audio, very large scale image and video datasets, and medical images with text records).The Visual Transformer will be a key component of next generation AI, able to address multiple downstream audio-visual tasks, significantly superseding the current limitations of computer vision systems, and enabling new and far reaching applications.A second objective addresses transfer and translation. We seek impact in a variety of other academic disciplines and industry which today greatly under-utilise the power of the latest computer vision ideas. We will target these disciplines to enable them to leapfrog the divide between what they use (or do not use) today which is dominated by manual review and highly interactive analysis frame-by-frame, to a new era where automated visual analytics of very large datasets becomes the norm. In short, our goal is to ensure that the newly developed methods are used by industry and academic researchers in other areas, and turned into products for societal and economic benefit. To this end open source software, datasets, and demonstrators will be disseminated on the project website.The ubiquity of digital images and videos means that every UK citizen may potentially benefit from the Programme research in different ways. One example is smart audio-visual glasses, that can pay attention to a person talking by using their lip movements to mask out other ambient sounds. A second is an app that can answer visual questions (or retrieve matches) for text-queries over large scale audio-visual collections, such as a person's entire personal videos. A third is AI-guided medical screening, that can aid a minimally trained healthcare professional to perform medical scans.

随着深度学习的出现和大数据的可用性，现在可以训练机器学习算法来完成大量的视觉任务，例如在云中标记个人图像集合、识别人脸和用手机进行3D形状扫描。然而，目前这些任务中的每一个都需要在一个非常大的图像数据集上训练神经网络，该数据集是专门为该任务收集和标记的。由此产生的网络是目标任务的优秀专家，但它们只理解训练期间经历的“封闭世界”，不能“说”任何有用的其他内容，也不能在没有重新训练的情况下应用于其他任务，也没有能力解释自己的决定或认识到自己的局限性。此外，目前的视觉算法通常是“单模态”的，它们对可能随时可用的其他模态（音频、文本）“充耳不闻”。该方案的核心目标是开发没有这些限制的下一代视听算法。我们将进行基础研究，以开发一种视觉转换器，该转换器能够进行视觉分析，具有人类视觉系统的灵活性和可解释性，并辅以其他“感官”-音频和文本。它将能够不断地从原始数据流中学习，而不需要为每个新任务对新数据集进行传统的“强监督”，并在多种数据类型（例如，带有音频的视频，非常大规模的图像和视频数据集，以及带有文本记录的医学图像）上传递和提取语义和几何信息。视觉转换器将成为下一代人工智能的关键组成部分，能够解决多个下游视听任务，显著取代当前计算机视觉系统的局限性，并实现新的和深远的应用。第二个目标涉及迁移和翻译。我们寻求对其他学科和行业的影响，这些学科和行业目前还没有充分利用最新计算机视觉思想的力量。我们将以这些学科为目标，使他们能够跨越他们今天使用（或不使用）的鸿沟，这些鸿沟主要是手工审查和高度交互的逐帧分析，进入一个新的时代，对非常大的数据集进行自动可视化分析成为常态。简而言之，我们的目标是确保新开发的方法被工业和学术研究人员在其他领域使用，并转化为具有社会和经济效益的产品。为此，开源软件、数据集和演示将在项目网站上发布。无处不在的数字图像和视频意味着每个英国公民都可能以不同的方式从该计划的研究中受益。智能视听眼镜就是一个例子，它可以通过一个人的嘴唇运动来掩盖其他环境声音，从而注意到一个人在说话。第二个是一个应用程序，它可以回答视觉问题（或检索匹配），用于大规模视听集合的文本查询，比如一个人的整个个人视频。第三种是人工智能引导的医疗筛查，它可以帮助受过最低限度训练的医疗保健专业人员进行医疗扫描。

项目成果

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

• DOI: 10.1109/iccv48922.2021.00175
• 发表时间: 2021-04
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Max Bain;Arsha Nagrani;Gül Varol;Andrew Zisserman

WhisperX: Time-Accurate Speech Transcription of Long-Form Audio

WhisperX：长格式音频的时间精确语音转录

• DOI: 10.21437/interspeech.2023-78
• 发表时间: 2023
• 作者: Bain M

Gaze-assisted automatic captioning of fetal ultrasound videos using three-way multi-modal deep neural networks

• DOI: 10.1016/j.media.2022.102630
• 发表时间: 2022-10-09
• 期刊: MEDICAL IMAGE ANALYSIS
• 影响因子: 10.9
• 作者: Alsharid, Mohammad;Cai, Yifan;Noble, J. Alison
• 通讯作者: Noble, J. Alison

PASS: An ImageNet replacement for self-supervised pretraining without human

PASS：ImageNet 替代自监督预训练，无需人工干预

• 发表时间: 2021
• 作者: Asano, Y

RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation

• DOI: 10.1109/cvpr52729.2023.01213
• 发表时间: 2022-11
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Titas Anciukevicius;Zexiang Xu;Matthew Fisher;Paul Henderson;Hakan Bilen;N. Mitra;Paul Guerrero