SI2-SSE: An Ecosystem of Reusable Image Analytics Pipelines

天文学已经进入了一个由望远镜和测量产生海量数据流的时代，这些数据流可以扫描几十年来电磁频谱中数万平方度的天空。这些新实验的前景--表征暗能量的性质和暗物质的组成，发现宇宙中能量最高的事件，追踪其轨道可能与地球轨道相交的小行星--只有在我们能够解决如何处理和分析这些天文调查每年产生的数十PB图像的挑战的情况下，才能实现。随着科学家收集越来越大的数据集的能力不断增强，通常是以图像的形式，我们的科学发现的潜力很快将受到限制，不是我们如何收集或存储数据，而是我们如何提取这些数据包含的知识(例如，我们如何解释数据中固有的噪声，以及了解我们何时检测到根本新的类别和有趣的事件或物理现象)。这个项目是为了开发一个开源的可伸缩框架，用于分析大型成像数据集。它旨在作为云服务运行，无缝整合新的或旧的图像处理算法，支持和优化复杂的分析工作流程，并将分析扩展到数千个处理器，而无需单个用户为特定的计算机平台或体系结构开发定制解决方案。这一框架将与为天文测量开发的最先进的图像分析算法相结合，以提供一个可供未来的望远镜和相机以及整个天文学社区使用的图像分析平台。除天文学外，该框架还将扩大，使利用成像数据的物理和生命科学(如神经科学、海洋学、生物学、地震学)的科学家能够把工作重点放在开发科学算法和分析上，而不是开发处理海量数据集所需的基础设施上。然而，这些技术和运行这些技术的系统的复杂性意味着，利用这些进步的用户数量很少；通常限于实验本身或一小群专家用户。正因为如此，社区作为一个整体并没有从天文学图像分析的重大投资中受益。在这个项目中，私人投资机构通过开发和部署一个可扩展的框架来分析小的和大的成像数据集来解决这些问题。这个基于云的系统将能够整合新的和传统的图像处理算法，支持和优化复杂的分析工作流程，将应用扩展到数千个处理器，而无需用户为特定平台开发定制代码，并支持用户之间高效地共享算法和分析结果。它将使广大天文学界能够使用最先进的图像分析算法(例如，为大型天气观测望远镜等测量开发的算法)，这样做将利用在开发这些技术上投入的数万小时。为了实现这一点，团队将从LSST数据分析管道中提取关键数据分析功能，并将其放入独立于LSST软件堆栈和数据访问机制的独立库中。他们将把这个库与Myria大数据管理系统整合在一起。Myria是一个弹性可扩展的大数据管理系统，作为我们在华盛顿大学开发的Amazon云中的一项服务运行。与其他大数据系统相比，Myria特别有吸引力，因为它将PostgreSQL数据库实例集成到其存储层中，从而提供了对PostgreSQL丰富的空间函数库的访问，这些函数库经常用于天文数据分析管道。同时，它还对新的和遗留的Python代码以及复杂的分析提供了丰富的支持。通过将LSST图像分析函数库与Myria集成，新的图像分析管道将变得非常容易编写。分析管道的框架将用MyriaL声明性查询语言(即，使用迭代等结构扩展的SQL)表示。核心数据处理函数将直接映射到Python函数，从而实现遗留代码的重用和轻松添加新函数。所产生的代码将服从于使用Myria服务的优化和高效执行。通过这样做，他们打算减少收养的障碍。用户将能够用Python表达他们的分析，而不必担心数据和计算将如何分布在集群中。*作为本提案的一部分开发的图像分析框架将作为开放源码软件公开提供。PIS将利用神经科学的使用案例来展示如何将他们为天文学开发的系统部署到多个领域。该项目得到了计算机与信息科学和工程局的高级数字基础设施办公室、天文科学司以及数学和物理科学局的多学科活动办公室的支持。

Astronomy has entered an era of massive data streams generated by telescopes and surveys that can scan tens of thousands of square degrees of the sky across many decades of the electromagnetic spectrum. The promise of these new experiments - characterizing the nature of dark energy and the composition of dark matter, discovering the most energetic events in the universe, tracking asteroids whose orbits may intersect with that of the Earth - will only be realized if we can address the challenge of how to process and analyze the tens of petabytes of images that these astronomical surveys will generate per year.  With the increasing capacity for scientists to collect ever larger sets of data, often in the form of images, our potential for scientific discovery will soon be limited not by how we collect or store data, but rather how we extract the knowledge that these data contain (e.g. how we account for noise inherent within the data, and understand when we have detected fundamentally new classes and interesting events or physical phenomena).  This project is to develop an open source scalable framework for the analysis of large imaging data sets. It is designed to operate as a cloud service, incorporate seamlessly new or legacy image processing algorithms, support and optimize complex analysis workflows, and scale analyses to thousands of processors without the need for an individual user to develop custom solutions for a specific computer platforms or architecture. This framework will be integrated with state-of-the-art image analysis algorithms developed for astronomical surveys  to provide an image analytics platform that can be used by future telescopes and cameras and the astronomical community as a whole. Beyond astronomy, the framework will be extended to enable scientists from the physical and life sciences that make use of imaging data (e.g. neuroscience, oceanography, biology, seismology) to focus their work on developing scientific algorithms and analyses rather than the infrastructure required to process massive data setsOver the last decade, there have been many advancements in astronomical image analysis algorithms and techniques; driven by new surveys and experiments. The complexity of these techniques and the systems that run them has, however, meant that the number of users who make use of these advancements is small; typically restricted to the experiments themselves or to a small group of expert users. Because of this, the community as a whole does not benefit from the significant investment in image analytics for astronomy.  In this project, the PIs address these issues by developing and deploying a scalable framework for the analysis of small and large imaging datasets. This cloud-based system will be able to incorporate new and legacy image processing algorithms, support and optimize complex analysis workflows, scale applications to thousands of processors without users needing to develop custom code for specific platforms, and support efficient sharing of algorithms and analysis results among users. It will enable state-of-the-art image analysis algorithms (e.g. those developed for surveys such as the Large Synoptic Survey Telescope; LSST) to be used by the broad astronomical community and in so doing will leverage then tens of thousands of hours that has been invested in the development of these techniques. To accomplish this the team will extract key data analysis functions from the LSST data analysis pipeline into a standalone library, independent of the LSST software stack and data access mechanisms.  They will integrate this library with the Myria big data management system. Myria is an elastically scalable big data management system that operates as a service in the Amazon cloud that wedeveloped at the University of Washington. Compared with other big data systems, Myria is especially attractive because it integrates PostgreSQL database instances within its storage layer and thus provides access to PostgreSQL's rich libraries of spatial functions, which are frequently used in astronomical data analysis pipelines. At the same time, it has rich support for new and legacy Python code and for complex analytics. By integrating the library of LSST image analytics functions with Myria, new image analytics pipelines will become significantly easier to write. The skeleton of the analysis pipeline will be expressed in the MyriaL declarative query language (i.e. SQL extended with constructs such as iterations and others). The core data processing functions will directly map to Python functions, enabling the reuse of legacy code and the easy addition of new functions. The resulting code will be amenable to optimization and efficient execution using the Myria service. By doing so, they intend to reduce barriers to adoption. Users will be able to express their analysis in Python without worrying about how data and computation will be distributed in a cluster.  The image analysis framework developed as part of this proposal will be made publicly available as open-source software. The PIs will utilize the use case of neuroscience to demonstrate how their system, developed for astronomy, can be deployed across multiple domains.This project is supported by the Office of Advanced Cyberinfrastructure in the Directorate for Computer & Information Science and Engineering, the Astronomical Sciences Division and Office of Multidisciplinary Activities in the Directorate of Mathematical and Physical Sciences.