Looking at People and Web-Scale Image Analysis

人类具有强烈的社交性，因此我们会花很多时间观察别人。简单地看一眼，我们就能感知姿势、动作、互动、意图和情绪，所有这些都是有效的社交互动所必需的。未来的计算机将会像我们一样花更多的时间观察人类。这对于设计新的人机界面、感知消费者设备、监控系统、视障人士辅助设备以及我们尚未设想的无数系统都很重要。****该提案旨在开发最先进的人体形状、外观、运动学和动力学模型，以促进基于视频的人体分析。这需要新的技术来估计人类的姿势、运动和与环境中物体的相互作用。核心技术有无数可能的应用，包括医疗保健，评估髋关节置换术后的运动恢复，评估高性能运动中的运动效率。****观察人，更广泛地说，视觉的一个关键是学习。视觉依赖于视觉世界的复杂模型（例如，物体形状，外观，运动等），这些模型构成了我们的先验信念，当图像本身只是部分限制场景时（大多数情况下是正确的），这些信念支持我们做出合理推断的能力。这些模型很难从基本原理中确定，所以从数据中学习通常更有效。这样做需要处理大量的图像或视频语料库，也就是大数据。****为了便于存储和分析庞大的图像语料库，我们正在开发新的数据结构和算法，用于快速建模、索引、搜索和检索高维数据。我们最近开发了几项关键技术，使我们能够在一台计算机上存储数十亿张图像，并在几毫秒内执行相似性搜索（查找与查询图像相似的数据库项）。我们计划继续改进和推广这些技术，允许人们从海量数据集中存储、检索、建模和学习（分类和回归）。这将有助于改善视觉方面的学习，它将在网络规模的搜索和检索应用中找到应用，比如根据图像内容搜索网络上的所有图像，但使用的计算资源只是谷歌等公司目前执行相同任务所需计算资源的一小部分

Humans are intensely social, and as such we spend a lot of time looking at people. With a simple glance we perceive pose, actions, interactions, intentions and emotions, all of which are necessary for effective social interaction. Future computers are going to spend as much time looking at people as we do. This will be important for the design of new man-machine interfaces, perceptive consumer devices, surveillance systems, aids for the visually impaired, and myriad systems we have yet to envision.****This proposal aims to develop state-of-the-art models of human shape, appearance, kinematics and dynamics to facilitate video-based analysis of people. This entails new techniques for estimating human pose, motion, and interactions with objects in the environment. The core technologies have myriad possible applications, including those in health care, assessing recovery of locomotion after hip replacement, to assessing the efficiency of movement in high performance sport.****One key to looking at people, and to vision more generally, is learning. Vision relies on sophisticated models of the visual world (e.g., object shape, appearance, motion, etc.), which comprise our prior beliefs that support our ability to make plausible inferences when the image alone only partially constrains the scene (which is true most of the time). And these models are so hard to specify from first principles, that it is often more effective to learn them from data. Doing so entails working with massive image or video corpora, a.k.a. big data.****To facilitate storage and analysis of huge image corpora, we are developing new data structures and algorithms for fast modeling, indexing, search and retrieval of high dimensional data. We recently developed several key technologies that allow us to store billions of images on a single computer, and perform similarity search (finding database items similar to a query image) in a few milliseconds. We plan to continue improving and generalizing such techniques, allowing one to store, retrieve, model and learn (classification and regression) from massive datasets. This will facilitate improved learning in vision, and it will find applications in web-scale search and retrieval applications, like searching all the images on the web based on image content, but using a small fraction of the computational resources that are currently required by companies like Google to perform the same tasks.**