ATD: Collaborative Research: Automatic, Adaptive Detection and Description of Change in Time-Lapse Imagery

该项目将提供自动、自适应检测和描述延时图像变化的算法--在很长一段时间内从同一场景获得的一系列图像。我们希望确定场景中何时发生“重大”变化，并用自然英语提供这些变化的文字描述，其中人类分析员提供反馈，以确定哪些变化是重要的(例如，正在建造的建筑、砍伐森林)或不重要的变化(例如，季节性变化)。我们将特别关注卫星或航空图像，对于这些图像，通常用于训练图像识别系统的数据集是不够的。这一基础研究有可能改变许多应用领域，包括监视、自主机器人、民用基础设施监测、高通量显微镜和气候科学，在所有这些领域，变化都是常见和重大的事件。我们在变化描述的新公式方面的工作也将影响到计算机视觉和自然语言处理的核心领域，在这些领域出现了许多类似的问题。该项目将包括研究生培训和博士后助理指导。发现变化是代理人感知世界和与世界互动的基本能力之一。用自然语言描述变化是使人类与这样一个代理人进行高效、准确和透明互动的关键。我们的工作将促进对这些目标的理论理解和实现这些目标的实际方法。具体地说，我们将解决在开发新的数学框架以定位渐进变化并用自然语言描述这些变化方面的上述挑战；我们将开发理论和实践手段来分析和克服观察到的图像中的腐败；我们将开发利用人类反馈的新理论和方法。这项工作将在变点检测和定位、使用深度神经网络和有限训练数据的图像重建以及适应人类反馈的多臂强盗方法等领域取得根本性进展。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project will provide algorithms for automatic, adaptive detection and description of changes in time-lapse imagery - a series of images obtained from the same scene over a long time frame. We wish to identify when there are "significant" changes in the scene, and provide a text description of those changes in natural English, where a human analyst provides feedback to determine what kinds of changes are important (e.g., a building being built, deforestation) or unimportant (e.g., seasonal changes). We will in particular focus on satellite or aerial imagery, for which data sets commonly used to train image recognition systems are inadequate. This fundamental research has the potential to transform many application domains, including surveillance, autonomous robotics, monitoring of civil infrastructure, high-throughput microscopy, and climate science, in all of which change is a common and significant occurrence. Our work on novel formulations of change description will also impact on core areas of computer vision and natural language processing, where many similar problems arise. The project will involve graduate students training and postdoctoral associate mentoring.Detecting change is one of the fundamental abilities for an agent perceiving and interacting with the world. Describing changes in natural language is key to making human interaction with such an agent efficient, accurate and transparent. Our work will advance both the theoretical understanding of these goals and the practical methods for implementing them. Specifically, we will address the above challenges for developing novel mathematical frameworks for localizing gradual changes and describing those changes in natural language; we will develop theoretical and practical means to analyze and overcome corruption in observed imagery; and we will develop novel theory and methods for leveraging human feedback. This work will yield fundamental advances in the fields of change point detection and localization, image reconstruction using deep neural networks and limited training data, and multi-armed bandit methodology for adapting to human feedback.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.