AXS - Enabling Analysis of Petascale Astronomical Datasets

随着大型、自动化、数字巡天的出现，天文学正迅速转变为一个pb级表格数据集正变得司空见惯的领域。不幸的是，这一增长并没有带来相应的工具和框架的改进：我们现在受到限制的不是数据集的丰富程度，而是无法从数据集中挖掘知识。当当今最具挑战性的问题需要对大量信息丰富的表格数据集进行重复、复杂的处理时，易于科学家使用的可扩展且稳定的工具至关重要。这是一个开发、打包和部署天文扩展Spark （AXS）的项目，AXS是一个可扩展的开源天文数据分析框架，建立在Apache Spark上。AXS将使天文学家，特别是那些不是数据管理专家的天文学家，能够使用行业标准工具设计和执行天文大数据分析。这将是一个变革性的提高，使社区能够从付出巨大代价收集的数据集中提取知识，从而释放它们在天文学所有领域的价值。工业界和学术界之间将有机会进行知识转移和建立伙伴关系。使用和创造的技术将在天文学课程中教授，这些课程材料将向公众公布。这将提高天文学学生在天文学以外的职业中的竞争力，并有助于培养一支具有全球竞争力的STEM劳动力队伍。AXS将使天文学家能够将他们的分析从个人笔记本电脑扩展到云或nsf支持的网络基础设施（CI）上的数千个节点。该系统已经原型化，并利用Spark，一个最先进的行业标准引擎，用于大数据处理，使查询和分析几乎任意大的天文目录成为可能，同时支持复杂的工作流程与天文学特定的操作。该工具将附带托管演示服务、文档，并支持在NSF CI资源和公共云平台上的部署。为了长期的可持续性，AXS将与主要利益相关者紧密合作，建立在开源工具和流程的基础上，并与天文学中广泛使用的AstroPy和PyData堆栈紧密集成。AXS还将健壮地扩展到大型计算集群，使nsf支持的和公共CI更容易被天文学家访问。该项目的发展将使其他工业和学术应用成为可能，特别是那些处理在一个球体上索引的大型表格式时空数据集的应用，例如地理空间分析。该奖项由美国国家科学基金会数学和物理科学理事会天文科学部颁发，并得到美国国家科学基金会高级网络基础设施办公室的联合支持。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Astronomy is being rapidly transformed by the advent of large, automated, digital sky surveys into a field where petabyte tabular data sets are becoming commonplace. Unfortunately, this increase has not been followed by commensurate improvements in the tools and frameworks: we are now limited not by the richness of our datasets, but by an inability to mine them for knowledge. When the most challenging questions of the day demand repeated, complex processing of large information-rich tabular datasets, scalable and stable tools that are easy to use by scientists are crucial. This is a project to develop, package, and deploy the Astronomical eXtensions for Spark (AXS), a scalable open-source astronomical data analysis framework built on Apache Spark. AXS will make it possible for astronomers, including and perhaps especially those who are not data management experts, to devise and execute astronomical big data analyses using industry-standard tools. This will be a transformative increase in the community's ability to extract knowledge from datasets collected at great expense, thus unlocking their value across all areas of astronomy. There will be opportunities for knowledge transfers and partnerships between industry and academia. The techniques used and created will be taught within the astronomy curriculum, and those curriculum materials will be made public. This will improve the competitiveness of astronomy students in careers beyond astronomy, and it will help to develop a globally competitive STEM workforce.AXS will enable astronomers to scale their analysis from a personal laptop to thousands of nodes on either cloud or NSF-supported cyberinfrastructure (CI). This system has already been prototyped, and leverages Spark, a state-of-the-art industry-standard engine for big data processing, to make it possible to query and analyze almost arbitrarily large astronomical catalogs while supporting complex workflows with astronomy-specific operations. The tool will be accompanied by a hosted demonstration service, documentation, and support for deployment on NSF CI resources and public cloud platforms. For long-term sustainability, AXS will be developed in a tight loop with major stakeholders, built on open source tools and processes, and strongly integrated with AstroPy and the PyData stack, which are widely used in astronomy. AXS will also robustly scale to large computational clusters, making both NSF-supported and public CI more accessible to astronomers. Developments by this project will enable other industrial and academic applications, especially those dealing with large, tabular, spatio-temporal datasets indexed on a sphere, such as geospatial analysis.This award by the Division of Astronomical Sciences within the NSF Directorate of Mathematical and Physical Sciences is jointly supported by the NSF Office of Advanced Cyberinfrastructure.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.