SI2-SSE: An Ecosystem of Reusable Image Analytics Pipelines

天文学已经进入了一个由望远镜和巡天产生的海量数据流的时代，这些数据流可以在几十年的电磁波谱中扫描数万平方度的天空。这些新实验的前景——表征暗能量的本质和暗物质的组成，发现宇宙中最具能量的事件，追踪轨道可能与地球相交的小行星——只有在我们能够解决如何处理和分析这些天文调查每年将产生的数十拍字节的图像的挑战时，才能实现。随着科学家收集越来越多的数据集（通常以图像的形式）的能力不断增强，我们科学发现的潜力很快就会受到限制，而不是我们如何收集或存储数据，而是我们如何提取这些数据中包含的知识（例如，我们如何解释数据中固有的噪音，以及当我们发现基本的新类别和有趣的事件或物理现象时，我们如何理解）。该项目旨在开发一个开源的可扩展框架，用于分析大型成像数据集。它被设计为作为云服务运行，无缝地整合新的或遗留的图像处理算法，支持和优化复杂的分析工作流程，并将分析扩展到数千个处理器，而无需单个用户为特定的计算机平台或架构开发定制解决方案。该框架将与为天文调查开发的最先进的图像分析算法集成，以提供一个图像分析平台，可用于未来的望远镜和相机以及整个天文学界。除了天文学之外，该框架还将得到扩展，使物理科学和生命科学领域的科学家能够利用成像数据（如神经科学、海洋学、生物学、地震学），将他们的工作重点放在开发科学算法和分析上，而不是处理大量数据集所需的基础设施。在新的调查和实验的推动下然而，这些技术和运行它们的系统的复杂性意味着使用这些先进技术的用户数量很少；通常仅限于实验本身或一小部分专家用户。正因为如此，整个社区并没有从天文图像分析的重大投资中受益。在这个项目中，pi通过开发和部署一个可扩展的框架来分析小型和大型成像数据集来解决这些问题。这种基于云的系统将能够整合新的和传统的图像处理算法，支持和优化复杂的分析工作流程，将应用程序扩展到数千个处理器，而无需用户为特定平台开发自定义代码，并支持用户之间算法和分析结果的有效共享。它将使最先进的图像分析算法（例如为大型综合巡天望远镜（LSST）等巡天而开发的算法）能够被广泛的天文学界使用，这样做将充分利用在这些技术开发上投入的数万小时。为了实现这一目标，团队将从LSST数据分析管道中提取关键数据分析功能到一个独立的库中，独立于LSST软件堆栈和数据访问机制。他们将把这个图书馆与Myria大数据管理系统整合在一起。Myria是一个弹性可扩展的大数据管理系统，在亚马逊云上作为服务运行，是我们在华盛顿大学开发的。与其他大数据系统相比，Myria特别有吸引力，因为它在其存储层中集成了PostgreSQL数据库实例，从而提供了对PostgreSQL丰富的空间函数库的访问，这些函数库经常用于天文数据分析管道。同时，它对新的和遗留的Python代码以及复杂的分析提供了丰富的支持。通过将LSST图像分析函数库与Myria集成，新的图像分析管道将变得更加容易编写。分析管道的框架将用MyriaL声明性查询语言表示（即使用迭代等结构扩展的SQL）。核心数据处理函数将直接映射到Python函数，从而可以重用遗留代码并轻松添加新函数。生成的代码可以使用Myria服务进行优化和高效执行。通过这样做，他们打算减少采用的障碍。用户将能够在Python中表达他们的分析，而不必担心数据和计算将如何在集群中分布。作为该提案的一部分开发的图像分析框架将作为开源软件公开提供。pi将利用神经科学的用例来演示他们为天文学开发的系统如何跨多个领域部署。该项目由计算机与信息科学与工程理事会高级网络基础设施办公室、天文科学部和数学与物理科学理事会多学科活动办公室提供支持。

Astronomy has entered an era of massive data streams generated by telescopes and surveys that can scan tens of thousands of square degrees of the sky across many decades of the electromagnetic spectrum. The promise of these new experiments - characterizing the nature of dark energy and the composition of dark matter, discovering the most energetic events in the universe, tracking asteroids whose orbits may intersect with that of the Earth - will only be realized if we can address the challenge of how to process and analyze the tens of petabytes of images that these astronomical surveys will generate per year.  With the increasing capacity for scientists to collect ever larger sets of data, often in the form of images, our potential for scientific discovery will soon be limited not by how we collect or store data, but rather how we extract the knowledge that these data contain (e.g. how we account for noise inherent within the data, and understand when we have detected fundamentally new classes and interesting events or physical phenomena).  This project is to develop an open source scalable framework for the analysis of large imaging data sets. It is designed to operate as a cloud service, incorporate seamlessly new or legacy image processing algorithms, support and optimize complex analysis workflows, and scale analyses to thousands of processors without the need for an individual user to develop custom solutions for a specific computer platforms or architecture. This framework will be integrated with state-of-the-art image analysis algorithms developed for astronomical surveys  to provide an image analytics platform that can be used by future telescopes and cameras and the astronomical community as a whole. Beyond astronomy, the framework will be extended to enable scientists from the physical and life sciences that make use of imaging data (e.g. neuroscience, oceanography, biology, seismology) to focus their work on developing scientific algorithms and analyses rather than the infrastructure required to process massive data setsOver the last decade, there have been many advancements in astronomical image analysis algorithms and techniques; driven by new surveys and experiments. The complexity of these techniques and the systems that run them has, however, meant that the number of users who make use of these advancements is small; typically restricted to the experiments themselves or to a small group of expert users. Because of this, the community as a whole does not benefit from the significant investment in image analytics for astronomy.  In this project, the PIs address these issues by developing and deploying a scalable framework for the analysis of small and large imaging datasets. This cloud-based system will be able to incorporate new and legacy image processing algorithms, support and optimize complex analysis workflows, scale applications to thousands of processors without users needing to develop custom code for specific platforms, and support efficient sharing of algorithms and analysis results among users. It will enable state-of-the-art image analysis algorithms (e.g. those developed for surveys such as the Large Synoptic Survey Telescope; LSST) to be used by the broad astronomical community and in so doing will leverage then tens of thousands of hours that has been invested in the development of these techniques. To accomplish this the team will extract key data analysis functions from the LSST data analysis pipeline into a standalone library, independent of the LSST software stack and data access mechanisms.  They will integrate this library with the Myria big data management system. Myria is an elastically scalable big data management system that operates as a service in the Amazon cloud that wedeveloped at the University of Washington. Compared with other big data systems, Myria is especially attractive because it integrates PostgreSQL database instances within its storage layer and thus provides access to PostgreSQL's rich libraries of spatial functions, which are frequently used in astronomical data analysis pipelines. At the same time, it has rich support for new and legacy Python code and for complex analytics. By integrating the library of LSST image analytics functions with Myria, new image analytics pipelines will become significantly easier to write. The skeleton of the analysis pipeline will be expressed in the MyriaL declarative query language (i.e. SQL extended with constructs such as iterations and others). The core data processing functions will directly map to Python functions, enabling the reuse of legacy code and the easy addition of new functions. The resulting code will be amenable to optimization and efficient execution using the Myria service. By doing so, they intend to reduce barriers to adoption. Users will be able to express their analysis in Python without worrying about how data and computation will be distributed in a cluster.  The image analysis framework developed as part of this proposal will be made publicly available as open-source software. The PIs will utilize the use case of neuroscience to demonstrate how their system, developed for astronomy, can be deployed across multiple domains.This project is supported by the Office of Advanced Cyberinfrastructure in the Directorate for Computer & Information Science and Engineering, the Astronomical Sciences Division and Office of Multidisciplinary Activities in the Directorate of Mathematical and Physical Sciences.