AXS - Enabling Analysis of Petascale Astronomical Datasets

天文学正在迅速转变的出现，大型，自动化，数字天空调查到一个领域，拍字节的表格数据集变得司空见惯。不幸的是，这一增长并没有伴随着工具和框架的相应改进：我们现在受到的限制不是数据集的丰富性，而是无法从中挖掘知识。 当当今最具挑战性的问题需要重复、复杂地处理信息丰富的大型表格数据集时，科学家们易于使用的可扩展和稳定的工具至关重要。这是一个开发、打包和部署Astronomical eXtensions for Spark（AXS）的项目，AXS是一个基于Apache Spark的可扩展开源天文数据分析框架。 AXS将使天文学家，包括可能特别是那些不是数据管理专家的人，能够使用行业标准工具设计和执行天文大数据分析。 这将是社区从以高昂代价收集的数据集中提取知识的能力的变革性增长，从而在天文学的所有领域释放其价值。 工业界和学术界之间将有机会进行知识转让和建立伙伴关系。 所使用和创造的技术将在天文学课程中教授，这些课程材料将公开。这将提高天文学学生在天文学以外职业的竞争力，并将有助于培养具有全球竞争力的STEM劳动力。AXS将使天文学家能够将他们的分析从个人笔记本电脑扩展到云或NSF支持的网络基础设施（CI）上的数千个节点。 该系统已经原型化，并利用Spark，一个最先进的行业标准的大数据处理引擎，使查询和分析几乎任意大的天文目录成为可能，同时支持复杂的工作流程与天文学特定的操作。 该工具将附带托管演示服务，文档以及对NSF CI资源和公共云平台上部署的支持。 为了长期可持续发展，AXS将与主要利益相关者紧密合作，建立在开源工具和流程的基础上，并与天文学中广泛使用的AstroPy和PyData堆栈紧密集成。 AXS还将强大地扩展到大型计算集群，使天文学家更容易获得NSF支持的和公共的CI。该项目的发展将使其他工业和学术应用，特别是那些处理大型，表格，时空数据集索引的领域，如地理空间分析。该奖项由NSF数学和物理科学理事会天文科学部颁发，并由NSF高级网络基础设施办公室共同支持。该奖项反映了NSF的法定使命，并被认为是值得的。通过使用基金会的知识价值和更广泛的影响审查标准进行评估来提供支持。

Astronomy is being rapidly transformed by the advent of large, automated, digital sky surveys into a field where petabyte tabular data sets are becoming commonplace. Unfortunately, this increase has not been followed by commensurate improvements in the tools and frameworks: we are now limited not by the richness of our datasets, but by an inability to mine them for knowledge. When the most challenging questions of the day demand repeated, complex processing of large information-rich tabular datasets, scalable and stable tools that are easy to use by scientists are crucial. This is a project to develop, package, and deploy the Astronomical eXtensions for Spark (AXS), a scalable open-source astronomical data analysis framework built on Apache Spark. AXS will make it possible for astronomers, including and perhaps especially those who are not data management experts, to devise and execute astronomical big data analyses using industry-standard tools. This will be a transformative increase in the community's ability to extract knowledge from datasets collected at great expense, thus unlocking their value across all areas of astronomy. There will be opportunities for knowledge transfers and partnerships between industry and academia. The techniques used and created will be taught within the astronomy curriculum, and those curriculum materials will be made public. This will improve the competitiveness of astronomy students in careers beyond astronomy, and it will help to develop a globally competitive STEM workforce.AXS will enable astronomers to scale their analysis from a personal laptop to thousands of nodes on either cloud or NSF-supported cyberinfrastructure (CI). This system has already been prototyped, and leverages Spark, a state-of-the-art industry-standard engine for big data processing, to make it possible to query and analyze almost arbitrarily large astronomical catalogs while supporting complex workflows with astronomy-specific operations. The tool will be accompanied by a hosted demonstration service, documentation, and support for deployment on NSF CI resources and public cloud platforms. For long-term sustainability, AXS will be developed in a tight loop with major stakeholders, built on open source tools and processes, and strongly integrated with AstroPy and the PyData stack, which are widely used in astronomy. AXS will also robustly scale to large computational clusters, making both NSF-supported and public CI more accessible to astronomers. Developments by this project will enable other industrial and academic applications, especially those dealing with large, tabular, spatio-temporal datasets indexed on a sphere, such as geospatial analysis.This award by the Division of Astronomical Sciences within the NSF Directorate of Mathematical and Physical Sciences is jointly supported by the NSF Office of Advanced Cyberinfrastructure.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.