Spatiotemporal Models for Analyzing and Understanding Video

视频包含用于理解和操纵周围世界的丰富信息；然而，作为一组逐点强度的原始形式，视频无法为进一步的处理提供明确的相关信息。拟议的研究计划旨在探索从原始视频流中提取有用信息的方法，以便进一步分析，从处理的最早阶段开始。在接下来的五年里，将审查以下三个基本调查问题。在视频流中自动识别人类行为的有用部分分解是什么？如何同时利用视频中的外观和动态信息来恢复成像场景的三维几何布局？观察到的密集个体的集体行为如何有助于跟踪每个个体？这三个基本问题将通过时间图像数学模型的紧密耦合、新算法的开发(例如图像处理)和使用受控合成数据和真实数据的系统经验评估来解决。这项拟议的研究在三个主要方面具有创新性。首先，该研究计划为我们对原始视频中可用信息的基本理解提供了广泛的理论见解。其次，这些见解为解决各种基于视觉的实际应用的算法的开发提供了坚实的基础。例如，动作识别可以用于视频索引和浏览，3D结构恢复是开发自主机器人的关键部分，在人口稠密的大型区域进行跟踪可以帮助安全和公共空间设计。第三，它让学生有机会发展他们在计算机视觉领域的技术和分析技能，更广泛地说，是新兴的数据科学领域。这种培训为学生在学术或工业研究和开发方面的职业生涯做好准备。

Video contains a wealth of information for understanding and manipulating the surrounding world; however, in its raw form as a set of pointwise intensities, video fails to make explicit pertinent information available for further processing. The proposed research program aims to explore methods to extract useful information from raw video streams for further analysis, starting at the earliest stages of processing. Over the next five years, the following three basic questions of inquiry will be examined. What makes a useful part decomposition of human actions for automatically recognizing them in a video stream? How can appearance and dynamic information in video be used simultaneously to recover the three dimensional geometric layout of an imaged scene? How can the observed collective behaviours of a densely packed ensemble of individuals aid in tracking each individual? These three basic questions will be addressed via a tight coupling of mathematical models of temporal imagery, the development of novel algorithms (e.g., image processing) and systematic empirical evaluation with both controlled synthetic and real data. The proposed research is innovative in three primary ways. First, the research program offers broad theoretical insights into our fundamental understanding of information available in raw video. Second, these insights provide the firm foundation for the development of algorithms addressing a variety of practical vision-based applications. For example, action recognition can be used for video indexing and browsing, 3D structure recovery is a key part of developing autonomous robots, and tracking in large densely populated areas can aid in security, safety and public space design. Third, it gives students the opportunity to develop their technical and analytical skills in the area of computer vision and more generally the emerging field of data science. This training prepares students for careers in academic or industrial research and development.