Looking at People and Web-Scale Image Analysis

人类具有强烈的社会性，因此我们花了很多时间观察人。通过简单的一瞥，我们就能感知到姿势、动作、互动、意图和情绪，所有这些都是有效的社交互动所必需的。未来的计算机将像我们一样花同样多的时间观察人。这对于设计新的人机界面、感知的消费设备、监控系统、为视障人士提供的辅助设备以及我们尚未设想的无数系统都将是重要的。 这项提议旨在开发最先进的人体形状、外表、运动学和动力学模型，以促进基于视频的人体分析。这就需要新的技术来估计人体姿势、运动以及与环境中物体的相互作用。这些核心技术有许多可能的应用，包括在医疗保健、评估髋关节置换后的运动恢复、评估高性能运动中的运动效率等。 看人，更广泛地看问题的关键之一，就是学习。视觉依赖于视觉世界的复杂模型(例如，对象形状、外观、运动等)，这些模型包含了我们的先前信念，这些信念支持我们在图像本身仅部分限制场景时做出可信的推断的能力(这在大多数情况下是正确的)。而且，这些模型很难从基本原理中确定，因此从数据中学习往往更有效。要做到这一点，需要处理大量的图像或视频语料库。大数据。 为了便于存储和分析海量图像语料库，我们正在开发新的数据结构和算法，用于快速建模、索引、搜索和检索高维数据。我们最近开发了几项关键技术，允许我们在一台计算机上存储数十亿张图像，并在几毫秒内执行相似性搜索(找到与查询图像相似的数据库项)。我们计划继续改进和推广这些技术，允许人们从海量数据集中存储、检索、建模和学习(分类和回归)。这将促进视觉学习的改进，它将在网络规模的搜索和检索应用程序中找到应用程序，比如基于图像内容搜索网络上的所有图像，但使用谷歌等公司目前执行相同任务所需的计算资源的一小部分。

Humans are intensely social, and as such we spend a lot of time looking at people. With a simple glance we perceive pose, actions, interactions, intentions and emotions, all of which are necessary for effective social interaction. Future computers are going to spend as much time looking at people as we do. This will be important for the design of new man-machine interfaces, perceptive consumer devices, surveillance systems, aids for the visually impaired, and myriad systems we have yet to envision. This proposal aims to develop state-of-the-art models of human shape, appearance, kinematics and dynamics to facilitate video-based analysis of people. This entails new techniques for estimating human pose, motion, and interactions with objects in the environment. The core technologies have myriad possible applications, including those in health care, assessing recovery of locomotion after hip replacement, to assessing the efficiency of movement in high performance sport. One key to looking at people, and to vision more generally, is learning. Vision relies on sophisticated models of the visual world (e.g., object shape, appearance, motion, etc.), which comprise our prior beliefs that support our ability to make plausible inferences when the image alone only partially constrains the scene (which is true most of the time). And these models are so hard to specify from first principles, that it is often more effective to learn them from data. Doing so entails working with massive image or video corpora, a.k.a. big data. To facilitate storage and analysis of huge image corpora, we are developing new data structures and algorithms for fast modeling, indexing, search and retrieval of high dimensional data. We recently developed several key technologies that allow us to store billions of images on a single computer, and perform similarity search (finding database items similar to a query image) in a few milliseconds. We plan to continue improving and generalizing such techniques, allowing one to store, retrieve, model and learn (classification and regression) from massive datasets. This will facilitate improved learning in vision, and it will find applications in web-scale search and retrieval applications, like searching all the images on the web based on image content, but using a small fraction of the computational resources that are currently required by companies like Google to perform the same tasks.