CAREER: Identifying Spatial and Dynamical Patterns from Images

从图像或视频中定量建模和自动提取语义信息（对象、动作和事件）一直是对系统自动化、人机界面、机器智能和信息技术感兴趣的科学家和工程师的一个困难和有趣的问题。这种困难和兴趣的基础很大程度上归因于视觉模式（形状或过程）的物理、空间和动态复杂性：光度学、几何和动力学中的高维性和固有的可变性只是这些特征中的一小部分。提出的研究计划旨在通过利用空间和动态模式中的不变属性来解决这种复杂性和可变性，这些模式可以建模为城市环境中常见的视觉模式的几何、动力学和物理中的离散和连续对称性的层次结构。来自光度学、多视角几何和系统理论的领域知识将用于分析建模和研究编码在视觉数据中的空间和动态对称性，并将开发有效的数值算法来直接从图像中检测这种对称性。因此，图像或视频的高级语义可以在对称层面上被机器更有效地识别、推断或学习。我们设想这样的建模范式将结合分析建模和统计推断技术的优点，并显著降低建模、分析和计算的复杂性。这将是成功开发高效、准确和强大的视觉系统的关键，该系统可用于识别城市环境中的各种物体、动作和事件。拟议研究计划的直接结果将是可扩展算法，该算法可以从图像或有效系统中自动生成三维几何模型，可以从大量视频输入中识别人类行为和事件。这些算法将极大地促进诸如安全监控、交通/环境监测、城市地区自动测绘、基于视觉的导航和自主机器人协调、即时体育报道/广播、电影编辑/视频索引以及医学图像分析等应用。随着认知科学和心理学对对称与感知之间关系的兴趣日益浓厚，本项目的结果也将有助于为生物和人工视觉感知的科学研究提供分析和计算基础。在教育方面，拟议的研究计划为开发新的跨学科课程提供了独特的机会，这些课程整合了科学方法、数学技能、计算技术和跨多个科学和工程学科的实验室实验，包括计算机视觉、系统理论和机器学习。这些课程将有助于改变和加强未来大学在机器人、机器视觉/学习和图像处理方面的工程教育。所要开发的图解例子和计算机程序也可以通过与常见的视觉经验和现象的联系，帮助向高中生或一般公众教授许多基本和重要的几何概念。

The quantitative modeling and automatic extraction of semantic information (objects, actions, and events) from images or videos has traditionally been a difficult and intriguing problem for scientists and engineers who are interested in system automation, human machine interface, machine intelligence, and information technologies. The basis for this difficulty, and interest, is largely attributed to physical, spatial, and dynamical complexity of visual patterns (of a shape or a process): high dimensionality and inherent variability in photometry, geometry, and dynamics are just a few of such characteristics. The proposed research program aims to tackle this complexity and variability by exploiting invariant properties in spatial and dynamical patterns that can be modeled as a hierarchy of discrete and continuous symmetries in the geometry, dynamics, and physics of visual patterns commonly encountered in an urban environment. Domain knowledge from photometry, multiple-view geometry, and systems theory will be used to analytically model and study both spatial and dynamical symmetries encoded in the visual data, and efficient numerical algorithms will be developed to detect such symmetries directly from images. High-level semantics of the images or videos can therefore be identified, inferred, or learned by machines more efficiently at the level of symmetries. We envision that such a modeling paradigm will marry the benefits of both analytical modeling and statistical inference techniques and significantly reduce the complexity in modeling, analysis, and computation. It will be the key to the success of developing efficient, accurate, and robust vision systems for identifying a wide range of objects, actions, and events in an urban environment. Direct outcome of the proposed research program will be scalable algorithms that can automatically generate three-dimensional geometric models from images or efficient systems that can identify in human actions and events from a large array of video input. Such algorithms will greatly facilitate applications such as security surveillance, traffic/environmental monitoring, automatic mapping of urban areas, vision-based navigation and coordination of autonomous robots, instant sports coverage/broadcast, movie edition/video indexing, and medical image analysis. Along with an increasing interest in the relations between symmetry and perception in cognitive science and psychology, the results from this program will also help provide an analytical and computational basis for the scientific study of biological and artificial visual perception in general. On the education end, the proposed research program provides unique opportunities for the development of new interdisciplinary courses that integrate scientific methods, mathematical skills, computational techniques, and laboratory experiments across multiple scientific and engineering disciplines, including computer vision, systems theory, and machine learning. These courses will help transform and enhance future college engineering education in robotics, machine vision/learning, and image processing. The illustrative examples and computer programs to be developed can also help teach many basic and important geometric concepts to high school students or to the general public through association with common visual experience and phenomena.