RI: Medium: Information Super-Resolution for Very Large Images

近年来，人工智能和机器学习已被应用于分析超大规模（VLS）图像，例如在航空或卫星图像和数字组织病理学分析中遇到的图像，以便领域科学家可以探索数据并形成新的假设。使用当前最先进的深度学习技术需要大量的详细注释（也称为标签）作为训练数据，这些注释可以与输入图像的大小成正比。因此，获取足够高分辨率的训练数据要么是不可能的，要么是非常昂贵的。在这个项目中，研究小组将开发一种方法，使用在小得多的低分辨率图像中收集的较弱（或辅助）信号来有效地约束高分辨率图像中标签的空间（或时间）统计分布。该框架大大减少了注释VLS图像的日常任务所需的人力，这对于预测环境趋势和癌症治疗结果的几个令人兴奋的应用程序至关重要。所开发的技术是通用的，它们的应用将在两个不同的领域进行演示，涉及非常大的图像，卫星图像和数字组织病理学。在环境应用方面，直接将卫星图像与政策相关指标（例如人口趋势、城市化、生物多样性丧失等）联系起来的能力将从根本上提高我们监测全球的能力。同样，能够从组织病理学的整张幻灯片图像中可靠地提取高分辨率信息，对于专注于开发新型诊断测试和众多精准医学应用的癌症研究(例如，患者分层、治疗选择、疾病进展、复发、治疗反应和无病生存，通过与临床、放射学、实验室、分子、药理学、结果数据)。该项目的技术目标是：i)研究团队通过跨分辨率匹配标签统计来解决超分辨率密集注释的问题。可微损失函数的一般方法将辅助约束映射到高分辨率标签。每个标签超分辨率损失是一个分布和一组统计值之间的可微距离度量；ii)研究团队将超分辨的概念推广到拓扑信息（通过持续同源），并使用多任务学习产生潜在表征，这些表征可以作为各种推理任务的基础；iii)在开发的框架中，研究团队对缺失辅助数据、异构辅助数据和同一区域的动态图像集进行建模，我们的损失可以很容易地集成到RNN/transformer架构和对抗学习范式中；iv)研究小组评估了两种增量人类参与的模式：1)向注释者展示他们的注释选择的效果，以帮助培养对高回报领域的直觉；2)基于强化学习的主动学习框架，模仿领域专家如何选择要标记的数据类型；v)研究团队通过一系列有充分根据的标签超分辨率应用来开发和评估想法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Artificial intelligence and Machine Learning have in recent years been applied to the analysis of very large scale (VLS) images such as those encountered in the analysis of aerial or satellite imagery and digital histopathology, so that domain scientists can explore the data and form novel hypotheses. The use of the current state-of-the-art deep learning techniques requires vast amounts of detailed annotations (a.k.a. labels) as training data, which can be proportional to the size of the input images. Thus, it is either impossible or very expensive to acquire enough high-resolution training data. In this project, the research team will develop a methodology that uses weaker (or auxiliary) signals collected in much smaller, low-resolution images to efficiently constrain the spatial (or temporal) statistical distribution of the labels in the high-resolution image. The framework significantly reduces the human effort needed for the mundane task of annotating VLS images, which is crucial for several exciting applications to predict environmental trends and cancer treatment outcomes. The developed techniques are general, and their application will be demonstrated in two different domains involving very large images, satellite imagery and digital histopathology. In environmental applications, the ability to directly connect satellite imagery to policy-relevant metrics of interest (e.g., population trends, urbanization, biodiversity loss, etc.) would radically improve our capacity to monitor the globe. Similarly, being able to reliably extract high resolution information from whole slide images of histopathology will be highly useful for cancer research focused on the development of novel diagnostic tests and numerous precision medicine applications (e.g., patient stratification, treatment selection, prediction of disease progression, recurrence, treatment response, and disease-free survival through downstream correlations with clinical, radiologic, laboratory, molecular, pharmacologic, and outcomes data). The technical aims of the project are: i) The research team addresses the problem of super-resolving dense annotations by matching label statistics across resolutions. The general methodology for differentiable loss functions maps auxiliary constraints to high-resolution labels. Each Label Super-Resolution loss is a differentiable distance metric between a distribution and a set of statistical values; ii) The research team generalizes the concept of super-resolution to topological information (through persistent homology) and use multi-task learning to produce latent representations that can be the basis of various inference tasks; iii) In the developed framework, the research team models missing auxiliary data, heterogeneous auxiliary data, and dynamic image sets of the same area and our losses can be easily integrated in RNN/transformer architectures and adversarial learning paradigms; iv) The research team evaluates two modalities of incremental human engagement: 1) Showing the annotator the effects of their annotation choices to help develop intuition for high return areas and 2) A reinforcement learning based active learning framework that imitates how domain experts select what kinds of data to label; and v) The research team develops and evaluates ideas through a number of well-grounded applications of Label Super-Resolution.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.