RI: Medium: Information Super-Resolution for Very Large Images

RI：中：超大图像的信息超分辨率

• 批准号: 2212046
• 负责人: Dimitrios Samaras
• 金额: $ 112.9万
• 依托单位: SUNY at Stony Brook
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2212046&HistoricalAwards=false
• 关键词: RI; Medium; Information; Super; Resolution

Artificial intelligence and Machine Learning have in recent years been applied to the analysis of very large scale (VLS) images such as those encountered in the analysis of aerial or satellite imagery and digital histopathology, so that domain scientists can explore the data and form novel hypotheses. The use of the current state-of-the-art deep learning techniques requires vast amounts of detailed annotations (a.k.a. labels) as training data, which can be proportional to the size of the input images. Thus, it is either impossible or very expensive to acquire enough high-resolution training data. In this project, the research team will develop a methodology that uses weaker (or auxiliary) signals collected in much smaller, low-resolution images to efficiently constrain the spatial (or temporal) statistical distribution of the labels in the high-resolution image. The framework significantly reduces the human effort needed for the mundane task of annotating VLS images, which is crucial for several exciting applications to predict environmental trends and cancer treatment outcomes. The developed techniques are general, and their application will be demonstrated in two different domains involving very large images, satellite imagery and digital histopathology. In environmental applications, the ability to directly connect satellite imagery to policy-relevant metrics of interest (e.g., population trends, urbanization, biodiversity loss, etc.) would radically improve our capacity to monitor the globe. Similarly, being able to reliably extract high resolution information from whole slide images of histopathology will be highly useful for cancer research focused on the development of novel diagnostic tests and numerous precision medicine applications (e.g., patient stratification, treatment selection, prediction of disease progression, recurrence, treatment response, and disease-free survival through downstream correlations with clinical, radiologic, laboratory, molecular, pharmacologic, and outcomes data). The technical aims of the project are: i) The research team addresses the problem of super-resolving dense annotations by matching label statistics across resolutions. The general methodology for differentiable loss functions maps auxiliary constraints to high-resolution labels. Each Label Super-Resolution loss is a differentiable distance metric between a distribution and a set of statistical values; ii) The research team generalizes the concept of super-resolution to topological information (through persistent homology) and use multi-task learning to produce latent representations that can be the basis of various inference tasks; iii) In the developed framework, the research team models missing auxiliary data, heterogeneous auxiliary data, and dynamic image sets of the same area and our losses can be easily integrated in RNN/transformer architectures and adversarial learning paradigms; iv) The research team evaluates two modalities of incremental human engagement: 1) Showing the annotator the effects of their annotation choices to help develop intuition for high return areas and 2) A reinforcement learning based active learning framework that imitates how domain experts select what kinds of data to label; and v) The research team develops and evaluates ideas through a number of well-grounded applications of Label Super-Resolution.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

近年来，人工智能和机器学习已被应用于超大规模（VLS）图像的分析，例如在航空或卫星图像和数字组织病理学分析中遇到的图像，以便领域科学家可以探索数据并形成新的假设。使用当前最先进的深度学习技术需要大量详细的注释（也称为注释）。标签）作为训练数据，其可以与输入图像的大小成比例。因此，获取足够的高分辨率训练数据是不可能的或非常昂贵的。在这个项目中，研究小组将开发一种方法，使用在小得多的低分辨率图像中收集的较弱（或辅助）信号来有效地约束高分辨率图像中标签的空间（或时间）统计分布。该框架显着减少了注释VLS图像的平凡任务所需的人力，这对于预测环境趋势和癌症治疗结果的几个令人兴奋的应用至关重要。所开发的技术是通用的，它们的应用将在两个不同的领域，涉及非常大的图像，卫星图像和数字组织病理学。 在环境应用中，将卫星图像直接与政策相关指标（例如，人口趋势、城市化、生物多样性丧失等）将从根本上提高我们监测地球仪的能力。类似地，能够从组织病理学的整个载玻片图像中可靠地提取高分辨率信息对于专注于开发新型诊断测试和许多精确医学应用（例如，患者分层、治疗选择、疾病进展预测、复发、治疗反应和通过与临床、放射学、实验室、分子学、药理学和结果数据的下游相关性的无病生存）。该项目的技术目标是：i）研究团队通过匹配不同分辨率的标签统计数据来解决超分辨率密集注释的问题。可微损失函数的一般方法将辅助约束映射到高分辨率标签。每个标签超分辨率损失是分布和一组统计值之间的可微距离度量; ii）研究团队将超分辨率的概念推广到拓扑信息（通过持久同源性）并使用多任务学习来产生潜在的表征，这些表征可以成为各种推理任务的基础; iii）在开发的框架中，研究团队对缺失的辅助数据，异构辅助数据，相同区域的动态图像集和我们的损失可以很容易地集成到RNN/Transformer架构和对抗学习范式中;研究团队评估了两种增量人类参与的模式：1）向注释者展示他们的注释选择的效果，以帮助开发对高回报领域的直觉; 2）基于强化学习的主动学习框架，模仿领域专家如何选择要标记的数据类型;和v）研究团队通过大量的标签超级应用程序开发和评估想法，决议：该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Token Sparsification for Faster Medical Image Segmentation

• DOI: 10.48550/arxiv.2303.06522
• 发表时间: 2023-03
• 期刊: Information processing in medical imaging : proceedings of the ... conference
• 作者: Lei Zhou;Huidong Liu;Joseph Bae;Junjun He;D. Samaras;P. Prasanna

Precise Location Matching Improves Dense Contrastive Learning in Digital Pathology

• DOI: 10.48550/arxiv.2212.12105
• 发表时间: 2022-12
• 期刊: ArXiv
• 作者: Jingwei Zhang;S. Kapse;Ke Ma;P. Prasanna;M. Vakalopoulou;J. Saltz;D. Samaras

Unsupervised Stain Decomposition via Inversion Regulation for Multiplex Immunohistochemistry Images

通过多重免疫组织化学图像的反转调节进行无监督染色分解

• 发表时间: 2023
• 期刊: Proceedings of Machine Learning Research
• 作者: Shahira Abousamra, Danielle Fassler

Topology-Guided Multi-Class Cell Context Generation for Digital Pathology

用于数字病理学的拓扑引导多类细胞上下文生成

• DOI: 10.1109/cvpr52729.2023.00324
• 发表时间: 2023
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR
• 作者: Abousamra, Shahira;Gupta, Rajarsi;Kurc, Tahsin;Samaras, Dimitris;Saltz, Joel;Chen, Chao
• 通讯作者: Chen, Chao

Zero-Shot Object Counting

• DOI: 10.1109/cvpr52729.2023.01492
• 发表时间: 2023-03
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Jingyi Xu;Hieu M. Le;Vu Nguyen;Viresh Ranjan;D. Samaras