ATD: Collaborative Research: Automatic, Adaptive Detection and Description of Change in Time-Lapse Imagery

该项目将提供自动、自适应检测和描述延时图像变化的算法，延时图像是在很长一段时间内从同一场景获得的一系列图像。我们希望确定场景中什么时候有“重大”变化，并以自然英语提供这些变化的文本描述，其中人类分析师提供反馈以确定哪些类型的变化是重要的（例如，正在建造的建筑物，森林砍伐）或不重要的（例如，季节变化）。我们将特别关注卫星或航空图像，通常用于训练图像识别系统的数据集是不够的。这项基础研究有可能改变许多应用领域，包括监视、自主机器人、民用基础设施监测、高通量显微镜和气候科学，在所有这些领域，变化都是常见和重要的。我们在变化描述的新公式方面的工作也将影响计算机视觉和自然语言处理的核心领域，在这些领域会出现许多类似的问题。该项目将包括研究生培训和博士后助理指导。感知变化是个体感知和与世界互动的基本能力之一。描述自然语言的变化是使人类与这种智能体的交互高效、准确和透明的关键。我们的工作将促进对这些目标的理论认识和实现这些目标的实际方法。具体来说，我们将解决上述挑战，开发新的数学框架来定位渐进变化并用自然语言描述这些变化；我们将发展理论和实践手段来分析和克服观测图像的腐败；我们将开发新的理论和方法来利用人类的反馈。这项工作将在变化点检测和定位、使用深度神经网络和有限训练数据的图像重建以及适应人类反馈的多臂强盗方法等领域取得根本性进展。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project will provide algorithms for automatic, adaptive detection and description of changes in time-lapse imagery - a series of images obtained from the same scene over a long time frame. We wish to identify when there are "significant" changes in the scene, and provide a text description of those changes in natural English, where a human analyst provides feedback to determine what kinds of changes are important (e.g., a building being built, deforestation) or unimportant (e.g., seasonal changes). We will in particular focus on satellite or aerial imagery, for which data sets commonly used to train image recognition systems are inadequate. This fundamental research has the potential to transform many application domains, including surveillance, autonomous robotics, monitoring of civil infrastructure, high-throughput microscopy, and climate science, in all of which change is a common and significant occurrence. Our work on novel formulations of change description will also impact on core areas of computer vision and natural language processing, where many similar problems arise. The project will involve graduate students training and postdoctoral associate mentoring.Detecting change is one of the fundamental abilities for an agent perceiving and interacting with the world. Describing changes in natural language is key to making human interaction with such an agent efficient, accurate and transparent. Our work will advance both the theoretical understanding of these goals and the practical methods for implementing them. Specifically, we will address the above challenges for developing novel mathematical frameworks for localizing gradual changes and describing those changes in natural language; we will develop theoretical and practical means to analyze and overcome corruption in observed imagery; and we will develop novel theory and methods for leveraging human feedback. This work will yield fundamental advances in the fields of change point detection and localization, image reconstruction using deep neural networks and limited training data, and multi-armed bandit methodology for adapting to human feedback.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.