Looking at People and Web-Scale Image Analysis

人类具有强烈的社会性，因此我们花了很多时间来观察人们。通过简单的一瞥，我们感知到姿势，动作，互动，意图和情绪，所有这些都是有效的社交互动所必需的。 未来的计算机将花和我们一样多的时间来观察人。 这对于设计新的人机界面，感知消费设备，监控系统，视障人士的辅助工具以及我们尚未设想的无数系统都很重要。该提案旨在开发最先进的人体形状、外观、运动学和动力学模型，以促进基于视频的人体分析。 这需要新的技术来估计人类的姿势，运动和与环境中的物体的交互。 核心技术有无数可能的应用，包括在医疗保健中的应用，评估髋关节置换术后运动的恢复，评估高性能运动中的运动效率。观察人的一个关键，以及更广泛的视觉，是学习。 视觉依赖于视觉世界的复杂模型（例如，对象形状、外观、运动等），它包括我们的先验信念，这些信念支持我们在图像仅部分约束场景时做出合理推断的能力（这在大多数情况下是正确的）。 这些模型很难从第一原理中指定，因此从数据中学习它们通常更有效。 这样做需要处理大量的图像或视频语料库。大数据。*为了便于存储和分析的巨大的图像语料库，我们正在开发新的数据结构和算法，快速建模，索引，搜索和检索的高维数据。我们最近开发了几项关键技术，使我们能够在一台计算机上存储数十亿张图像，并在几毫秒内执行相似性搜索（找到与查询图像相似的数据库项）。 我们计划继续改进和推广这些技术，允许人们从海量数据集中存储，检索，建模和学习（分类和回归）。 这将有助于改善视觉学习，并将在网络规模的搜索和检索应用中找到应用，例如基于图像内容搜索网络上的所有图像，但使用的计算资源仅为Google等公司目前执行相同任务所需的一小部分。

Humans are intensely social, and as such we spend a lot of time looking at people. With a simple glance we perceive pose, actions, interactions, intentions and emotions, all of which are necessary for effective social interaction. Future computers are going to spend as much time looking at people as we do. This will be important for the design of new man-machine interfaces, perceptive consumer devices, surveillance systems, aids for the visually impaired, and myriad systems we have yet to envision.****This proposal aims to develop state-of-the-art models of human shape, appearance, kinematics and dynamics to facilitate video-based analysis of people. This entails new techniques for estimating human pose, motion, and interactions with objects in the environment. The core technologies have myriad possible applications, including those in health care, assessing recovery of locomotion after hip replacement, to assessing the efficiency of movement in high performance sport.****One key to looking at people, and to vision more generally, is learning. Vision relies on sophisticated models of the visual world (e.g., object shape, appearance, motion, etc.), which comprise our prior beliefs that support our ability to make plausible inferences when the image alone only partially constrains the scene (which is true most of the time). And these models are so hard to specify from first principles, that it is often more effective to learn them from data. Doing so entails working with massive image or video corpora, a.k.a. big data.****To facilitate storage and analysis of huge image corpora, we are developing new data structures and algorithms for fast modeling, indexing, search and retrieval of high dimensional data. We recently developed several key technologies that allow us to store billions of images on a single computer, and perform similarity search (finding database items similar to a query image) in a few milliseconds. We plan to continue improving and generalizing such techniques, allowing one to store, retrieve, model and learn (classification and regression) from massive datasets. This will facilitate improved learning in vision, and it will find applications in web-scale search and retrieval applications, like searching all the images on the web based on image content, but using a small fraction of the computational resources that are currently required by companies like Google to perform the same tasks.**