AXS - Enabling Analysis of Petascale Astronomical Datasets

AXS - 支持千万亿级天文数据集的分析

• 批准号: 2003196
• 负责人: Mario Juric
• 金额: $ 57.93万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-08-01 至 2024-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2003196&HistoricalAwards=false
• 关键词: AXS; Enabling; Analysis; Petascale; Astronomical

Astronomy is being rapidly transformed by the advent of large, automated, digital sky surveys into a field where petabyte tabular data sets are becoming commonplace. Unfortunately, this increase has not been followed by commensurate improvements in the tools and frameworks: we are now limited not by the richness of our datasets, but by an inability to mine them for knowledge. When the most challenging questions of the day demand repeated, complex processing of large information-rich tabular datasets, scalable and stable tools that are easy to use by scientists are crucial. This is a project to develop, package, and deploy the Astronomical eXtensions for Spark (AXS), a scalable open-source astronomical data analysis framework built on Apache Spark. AXS will make it possible for astronomers, including and perhaps especially those who are not data management experts, to devise and execute astronomical big data analyses using industry-standard tools. This will be a transformative increase in the community's ability to extract knowledge from datasets collected at great expense, thus unlocking their value across all areas of astronomy. There will be opportunities for knowledge transfers and partnerships between industry and academia. The techniques used and created will be taught within the astronomy curriculum, and those curriculum materials will be made public. This will improve the competitiveness of astronomy students in careers beyond astronomy, and it will help to develop a globally competitive STEM workforce.AXS will enable astronomers to scale their analysis from a personal laptop to thousands of nodes on either cloud or NSF-supported cyberinfrastructure (CI). This system has already been prototyped, and leverages Spark, a state-of-the-art industry-standard engine for big data processing, to make it possible to query and analyze almost arbitrarily large astronomical catalogs while supporting complex workflows with astronomy-specific operations. The tool will be accompanied by a hosted demonstration service, documentation, and support for deployment on NSF CI resources and public cloud platforms. For long-term sustainability, AXS will be developed in a tight loop with major stakeholders, built on open source tools and processes, and strongly integrated with AstroPy and the PyData stack, which are widely used in astronomy. AXS will also robustly scale to large computational clusters, making both NSF-supported and public CI more accessible to astronomers. Developments by this project will enable other industrial and academic applications, especially those dealing with large, tabular, spatio-temporal datasets indexed on a sphere, such as geospatial analysis.This award by the Division of Astronomical Sciences within the NSF Directorate of Mathematical and Physical Sciences is jointly supported by the NSF Office of Advanced Cyberinfrastructure.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

随着大规模、自动化、数字化天空测量的出现，天文学正在迅速转变为一个以PB为单位的表格数据集正变得司空见惯的领域。不幸的是，在这一增长之后，工具和框架并没有相应的改进：我们现在受到的限制不是我们数据集的丰富性，而是无法从它们中挖掘知识。当当今最具挑战性的问题需要对信息丰富的大型表格数据集进行重复、复杂的处理时，科学家易于使用的可扩展和稳定的工具至关重要。这是一个开发、打包和部署Astronomical Expansion for Spark(AXS)的项目，AXS是一个构建在ApacheSpark上的可伸缩开源天文数据分析框架。AXS将使天文学家，包括可能尤其是那些不是数据管理专家的天文学家，使用行业标准工具设计和执行天文大数据分析。这将是社区从以巨大代价收集的数据集中提取知识的能力的革命性增长，从而释放它们在天文学所有领域的价值。将有机会在产业界和学术界之间进行知识转移和建立伙伴关系。使用和创造的技术将在天文学课程中教授，这些课程材料将公之于众。这将提高天文学学生在天文学以外的职业中的竞争力，并将有助于发展一支具有全球竞争力的STEM工作队伍。AXS将使天文学家能够将他们的分析从个人笔记本电脑扩展到云或NSF支持的网络基础设施(CI)上的数千个节点。该系统已经制作了原型，并利用Spark这一用于大数据处理的最先进的行业标准引擎，使查询和分析几乎任意大的天文星表成为可能，同时支持具有天文特定操作的复杂工作流。该工具将附带托管演示服务、文档和在NSF CI资源和公共云平台上部署的支持。为了长期可持续发展，AXS将与主要利益攸关方在一个紧密的循环中开发，建立在开源工具和流程的基础上，并与天文学中广泛使用的AstroPy和PyData堆栈紧密集成。AXS还将强劲地扩展到大型计算集群，使天文学家更容易获得NSF支持的和公共CI。该项目的发展将使其他工业和学术应用成为可能，特别是那些处理在球体上索引的大型表格时空数据集的应用，例如地理空间分析。该奖项由NSF数学和物理科学局内的天文科学部联合支持。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

A Scalable Cloud-Based Analysis Platform for Survey Astronomy

用于巡天天文学的可扩展的基于云的分析平台

• 发表时间: 2020
• 期刊: Gateways 2020
• 作者: Stetzler, S;Slater, C;Zecevic, P;Juric, M.
• 通讯作者: Juric, M.

The Astronomy Commons Platform: A Deployable Cloud-based Analysis Platform for Astronomy

• DOI: 10.3847/1538-3881/ac77fb
• 发表时间: 2022-06
• 期刊: The Astronomical Journal
• 作者: S. Stetzler;M. Juri'c;Kyle Boone;Andrew J. Connolly;C. Slater;Petar Zevcevi'c