SI2-SSE: An Ecosystem of Reusable Image Analytics Pipelines

SI2-SSE：可重用图像分析管道生态系统

• 批准号: 1739419
• 负责人: Andrew Connolly
• 金额: $ 50万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-01 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1739419&HistoricalAwards=false
• 关键词: SI2; SSE; Ecosystem; Reusable; Image

Astronomy has entered an era of massive data streams generated by telescopes and surveys that can scan tens of thousands of square degrees of the sky across many decades of the electromagnetic spectrum. The promise of these new experiments - characterizing the nature of dark energy and the composition of dark matter, discovering the most energetic events in the universe, tracking asteroids whose orbits may intersect with that of the Earth - will only be realized if we can address the challenge of how to process and analyze the tens of petabytes of images that these astronomical surveys will generate per year.  With the increasing capacity for scientists to collect ever larger sets of data, often in the form of images, our potential for scientific discovery will soon be limited not by how we collect or store data, but rather how we extract the knowledge that these data contain (e.g. how we account for noise inherent within the data, and understand when we have detected fundamentally new classes and interesting events or physical phenomena).  This project is to develop an open source scalable framework for the analysis of large imaging data sets. It is designed to operate as a cloud service, incorporate seamlessly new or legacy image processing algorithms, support and optimize complex analysis workflows, and scale analyses to thousands of processors without the need for an individual user to develop custom solutions for a specific computer platforms or architecture. This framework will be integrated with state-of-the-art image analysis algorithms developed for astronomical surveys  to provide an image analytics platform that can be used by future telescopes and cameras and the astronomical community as a whole. Beyond astronomy, the framework will be extended to enable scientists from the physical and life sciences that make use of imaging data (e.g. neuroscience, oceanography, biology, seismology) to focus their work on developing scientific algorithms and analyses rather than the infrastructure required to process massive data setsOver the last decade, there have been many advancements in astronomical image analysis algorithms and techniques; driven by new surveys and experiments. The complexity of these techniques and the systems that run them has, however, meant that the number of users who make use of these advancements is small; typically restricted to the experiments themselves or to a small group of expert users. Because of this, the community as a whole does not benefit from the significant investment in image analytics for astronomy.  In this project, the PIs address these issues by developing and deploying a scalable framework for the analysis of small and large imaging datasets. This cloud-based system will be able to incorporate new and legacy image processing algorithms, support and optimize complex analysis workflows, scale applications to thousands of processors without users needing to develop custom code for specific platforms, and support efficient sharing of algorithms and analysis results among users. It will enable state-of-the-art image analysis algorithms (e.g. those developed for surveys such as the Large Synoptic Survey Telescope; LSST) to be used by the broad astronomical community and in so doing will leverage then tens of thousands of hours that has been invested in the development of these techniques. To accomplish this the team will extract key data analysis functions from the LSST data analysis pipeline into a standalone library, independent of the LSST software stack and data access mechanisms.  They will integrate this library with the Myria big data management system. Myria is an elastically scalable big data management system that operates as a service in the Amazon cloud that wedeveloped at the University of Washington. Compared with other big data systems, Myria is especially attractive because it integrates PostgreSQL database instances within its storage layer and thus provides access to PostgreSQL's rich libraries of spatial functions, which are frequently used in astronomical data analysis pipelines. At the same time, it has rich support for new and legacy Python code and for complex analytics. By integrating the library of LSST image analytics functions with Myria, new image analytics pipelines will become significantly easier to write. The skeleton of the analysis pipeline will be expressed in the MyriaL declarative query language (i.e. SQL extended with constructs such as iterations and others). The core data processing functions will directly map to Python functions, enabling the reuse of legacy code and the easy addition of new functions. The resulting code will be amenable to optimization and efficient execution using the Myria service. By doing so, they intend to reduce barriers to adoption. Users will be able to express their analysis in Python without worrying about how data and computation will be distributed in a cluster.  The image analysis framework developed as part of this proposal will be made publicly available as open-source software. The PIs will utilize the use case of neuroscience to demonstrate how their system, developed for astronomy, can be deployed across multiple domains.This project is supported by the Office of Advanced Cyberinfrastructure in the Directorate for Computer & Information Science and Engineering, the Astronomical Sciences Division and Office of Multidisciplinary Activities in the Directorate of Mathematical and Physical Sciences.

天文学已经进入了一个由望远镜和调查产生的大量数据流的时代，这些数据流可以在几十年的电磁频谱中扫描数万平方度的天空。这些新实验的承诺-表征暗能量的性质和暗物质的组成，发现宇宙中最具活力的事件，跟踪轨道可能与地球相交的小行星-只有当我们能够解决如何处理和分析这些天文调查每年将产生的数十PB图像的挑战时才能实现。 随着科学家收集更大数据集（通常以图像形式）的能力不断提高，我们的科学发现潜力很快将不再受到如何收集或存储数据的限制，而是受到如何提取这些数据所包含的知识的限制。（例如，我们如何解释数据中固有的噪声，以及何时发现了全新的类别和有趣的事件或物理现象）。 该项目旨在开发一个开源的可扩展框架，用于分析大型成像数据集。它旨在作为云服务运行，无缝整合新的或传统的图像处理算法，支持和优化复杂的分析工作流程，并将分析扩展到数千个处理器，而无需单个用户为特定的计算机平台或架构开发定制解决方案。该框架将与为天文测量开发的最先进的图像分析算法相结合，以提供一个可供未来望远镜和相机以及整个天文学界使用的图像分析平台。除了天文学之外，该框架还将扩展到使物理和生命科学的科学家能够利用成像数据（如神经科学、海洋学、生物学、地震学）将工作重点放在开发科学算法和分析上，而不是处理大量数据集所需的基础设施上。由新的调查和实验驱动。然而，这些技术和运行它们的系统的复杂性意味着利用这些进步的用户数量很少;通常仅限于实验本身或一小群专家用户。正因为如此，整个社区并没有从天文学图像分析的重大投资中受益。 在该项目中，PI通过开发和部署用于分析小型和大型成像数据集的可扩展框架来解决这些问题。这个基于云的系统将能够整合新的和传统的图像处理算法，支持和优化复杂的分析工作流程，将应用程序扩展到数千个处理器，而无需用户为特定平台开发自定义代码，并支持用户之间高效共享算法和分析结果。它将使最先进的图像分析算法（例如，为大型综合巡天望远镜（LSST）等巡天而开发的算法）能够被广大的天文学界所使用，这样做将利用在开发这些技术方面投入的数万小时。为了实现这一目标，该团队将从LSST数据分析管道中提取关键数据分析功能到独立的库中，独立于LSST软件堆栈和数据访问机制。 他们将把这个图书馆与Myria大数据管理系统集成在一起。Myria是一个可弹性扩展的大数据管理系统，它作为我们在华盛顿大学开发的亚马逊云服务运行。与其他大数据系统相比，Myria特别有吸引力，因为它在其存储层中集成了PostgreSQL数据库实例，从而提供了对PostgreSQL丰富的空间函数库的访问，这些空间函数库经常用于天文数据分析管道。同时，它对新的和旧的Python代码以及复杂的分析提供了丰富的支持。通过将LSST图像分析函数库与Myria集成，新的图像分析管道将变得更加容易编写。分析管道的骨架将用MyriaL声明性查询语言（即，使用迭代等构造扩展的SQL）表示。核心数据处理函数将直接映射到Python函数，从而可以重用遗留代码并轻松添加新函数。生成的代码将易于使用Myria服务进行优化和高效执行。通过这样做，他们打算减少采用的障碍。用户将能够用Python表达他们的分析，而不用担心数据和计算将如何在集群中分布。 作为该提案一部分开发的图像分析框架将作为开放源码软件公开提供。PI将利用神经科学的用例来演示他们为天文学开发的系统如何在多个领域部署。该项目得到了计算机信息科学与工程局高级网络基础设施办公室、天文科学司和数学与物理科学局多学科活动办公室的支持。&

项目成果

Sub-band Image Reconstruction Using Differential Chromatic Refraction

使用差分色折射的子带图像重建

• DOI: 10.3847/1538-3881/ab139f
• 发表时间: 2019
• 期刊: The Astronomical Journal
• 作者: Lee, Matthias A.;Budavári, Tamás;Sullivan, Ian S.;Connolly, Andrew J.
• 通讯作者: Connolly, Andrew J.

Fast Algorithms for Slow Moving Asteroids: Constraints on the Distribution of Kuiper Belt Objects

• DOI: 10.3847/1538-3881/aafd2d
• 发表时间: 2019-01
• 期刊: The Astronomical Journal
• 作者: Peter J. Whidden;J. Kalmbach;A. Connolly;R. L. Jones;H. Smotherman;D. Bektešević;C. Slater;A. Becker;Ž. Ivezić;Mario Juri'c;B. Bolin;Joachim Moeyens;F. Förster;V. Golkhou

Sifting through the Static: Moving Object Detection in Difference Images

筛选静态：差异图像中的运动物体检测

• DOI: 10.3847/1538-3881/ac22ff
• 发表时间: 2021
• 期刊: The Astronomical Journal
• 作者: Smotherman, Hayden;Connolly, Andrew J.;Kalmbach, J. Bryce;Portillo, Stephen K.;Bektesevic, Dino;Eggl, Siegfried;Juric, Mario;Moeyens, Joachim;Whidden, Peter J.
• 通讯作者: Whidden, Peter J.

Toward Sampling for Deep Learning Model Diagnosis

• DOI: 10.1109/icde48307.2020.00201
• 发表时间: 2020-04
• 期刊: 2020 IEEE 36th International Conference on Data Engineering (ICDE)
• 作者: Parmita Mehta;S. Portillo;M. Balazinska;Andrew J. Connolly

A Gateway to Astronomical Image Processing: Vera C. RubinObservatory LSST Science Pipelines on AWS

天文图像处理的门户：AWS 上的 Vera C. RubinObservatory LSST Science Pipelines

• 发表时间: 2020
• 期刊: Online
• 作者: Bektesevic, Dino;Chiang, Hsin-Fang;Lim, Kian-Tat;Miller, Todd L.;Thain, Greg;Jenness, Tim;Bosch, James;Salnikov, Andrei;Connolly, Andrew
• 通讯作者: Connolly, Andrew