Seebibyte: Visual Search for the Era of Big Data

Seebibyte：大数据时代的视觉搜索

• 批准号: EP/M013774/1
• 负责人: Andrew Zisserman
• 金额: $ 569.27万
• 依托单位: University of Oxford
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2015
• 资助国家: 英国
• 起止时间: 2015 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FM013774%2F1
• 关键词: Seebibyte; Visual; Search; Era; Big

The Programme is organised into two themes. Research theme one will develop new computer vision algorithms to enable efficient search and description of vast image and video datasets - for example of the entire video archive of the BBC. Our vision is that anything visual should be searchable for, in the manner of a Google search of the web: by specifying a query, and having results returned immediately, irrespective of the size of the data. Such enabling capabilities will have widespread application both for general image/video search - consider how Google's web search has opened up new areas - and also for designing customized solutions for searching.A second aspect of theme 1 is to automatically extract detailed descriptions of the visual content. The aim here is to achieve human like performance and beyond, for example in recognizing configurations of parts and spatial layout, counting and delineating objects, or recognizing human actions and inter-actions in videos, significantly superseding the current limitations of computer vision systems, and enabling new and far reaching applications. The new algorithms will learn automatically, building on recent breakthroughs in large scale discriminative and deep machine learning. They will be capable of weakly-supervised learning, for example from images and videos downloaded from the internet, and require very little human supervision.The second theme addresses transfer and translation. This also has two aspects. The first is to apply the new computer vision methodologies to `non-natural' sensors and devices, such as ultrasound imaging and X-ray, which have different characteristics (noise, dimension, invariances) to the standard RGB channels of data captured by `natural' cameras (iphones, TV cameras). The second aspect of this theme is to seek impact in a variety of other disciplines and industry which today greatly under-utilise the power of the latest computer vision ideas. We will target these disciplines to enable them to leapfrog the divide between what they use (or do not use) today which is dominated by manual review and highly interactive analysis frame-by-frame, to a new era where automated efficient sorting, detection and mensuration of very large datasets becomes the norm. In short, our goal is to ensure that the newly developed methods are used by academic researchers in other areas, and turned into products for societal and economic benefit. To this end open source software, datasets, and demonstrators will be disseminated on the project website.The ubiquity of digital imaging means that every UK citizen may potentially benefit from the Programme research in different ways. One example is an enhanced iplayer that can search for where particular characters appear in a programme, or intelligently fast forward to the next `hugging' sequence. A second is wider deployment of lower cost imaging solutions in healthcare delivery. A third, also motivated by healthcare, is through the employment of new machine learning methods for validating targets for drug discovery based on microscopy images

该方案分为两个主题。研究主题一将开发新的计算机视觉算法，以实现对大量图像和视频数据集的有效搜索和描述-例如BBC的整个视频档案。我们的愿景是，任何可视化的东西都应该是可搜索的，就像谷歌搜索网络一样：通过指定一个查询，并立即返回结果，而不管数据的大小。这种使能能力将广泛应用于一般的图像/视频搜索（考虑Google的网络搜索如何开辟新的领域）以及设计用于搜索的定制解决方案。主题1的第二个方面是自动提取视觉内容的详细描述。其目标是实现类似人类的性能，例如识别部件和空间布局的配置，计数和描绘对象，或识别视频中的人类动作和交互，大大取代计算机视觉系统的当前限制，并实现新的和深远的应用。新算法将自动学习，建立在大规模判别和深度机器学习的最新突破基础上。它们将能够进行弱监督学习，例如从互联网上下载的图像和视频，并且几乎不需要人工监督。第二个主题涉及转移和翻译。这也有两个方面。首先是将新的计算机视觉方法应用于“非自然”传感器和设备，如超声成像和X射线，这些传感器和设备与“自然”相机（iPhone，电视摄像机）捕获的标准RGB通道数据具有不同的特性（噪声，尺寸，不变性）。这个主题的第二个方面是寻求对其他各种学科和行业的影响，这些学科和行业目前大大低估了最新计算机视觉思想的力量。我们将针对这些学科，使他们能够跨越他们今天使用（或不使用）的内容之间的鸿沟，这些内容由手动审查和高度交互式的逐帧分析主导，进入一个新时代，自动化的高效排序，检测和测量非常大的数据集成为常态。简而言之，我们的目标是确保新开发的方法被其他领域的学术研究人员使用，并转化为具有社会和经济效益的产品。为此，将在项目网站上发布开源软件、数据集和演示程序。数字成像的普遍性意味着每个英国公民都可能以不同的方式从该计划的研究中受益。一个例子是一个增强的iplayer，它可以搜索特定角色在节目中出现的位置，或者智能地快进到下一个“拥抱”序列。第二个是在医疗保健服务中更广泛地部署低成本成像解决方案。第三个也是出于医疗保健的动机，是通过采用新的机器学习方法来验证基于显微图像的药物发现目标

项目成果

Deep Audio-Visual Speech Recognition

深度视听语音识别

• DOI: 10.48550/arxiv.1809.02108
• 发表时间: 2018
• 作者: Afouras T

Efficient Linear Programming for Dense CRFs

• DOI: 10.1109/cvpr.2017.313
• 发表时间: 2016-11
• 期刊: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Thalaiyasingam Ajanthan;Alban Desmaison;Rudy Bunel;M. Salzmann;Philip H. S. Torr;M. P. Kumar

The Conversation: Deep Audio-Visual Speech Enhancement

• DOI: 10.21437/interspeech.2018-1400
• 发表时间: 2018-04
• 期刊: ArXiv
• 作者: Triantafyllos Afouras;Joon Son Chung;Andrew Zisserman

Now You're Speaking My Language: Visual Language Identification

现在你正在说我的语言：视觉语言识别

• DOI: 10.21437/interspeech.2020-2921
• 发表时间: 2020
• 作者: Afouras T

My lips are concealed: Audio-visual speech enhancement through obstructions

• DOI: 10.21437/interspeech.2019-3114
• 发表时间: 2019-07
• 作者: Triantafyllos Afouras;Joon Son Chung;Andrew Zisserman