Efficient Astronomical Data Processing Among Distributed Astronomical Radio Observatories

天文学已经成为一门数据丰富的科学，也是计算资源的最大消费者之一。NSF的国家射电天文台(NRAO)收集和处理来自广泛选择的设施的射电望远镜数据，但这种集中的方法无法提供及时的分析，这推动了对分散的天文台之间对高效天文数据处理系统的日益增长的需求。随着未来的望远镜项目上线，数据量将急剧增长，天文实验将需要更准确的数据分析和处理，以及更详细的结果。一个关键的挑战是作业调度，以最大化应用程序性能并最大限度地降低系统成本。该项目将为地理分布式观测站设计和开发系统软件，重点是天文数据处理性能测量和性能分析，以及高效的处理系统设计。研究成果将通过开放源码软件发布来传播。该项目将在相关课程中展示，并将为参与的本科生、研究生、K-12学生和教职员工以及代表不足的学生和女性学生提供全面的培训和协作研究机会。该软件用于可扩展和高效的天文(SEA)数据处理，包括作业调度、作业有向无环图(DAG)配置和全球数据分布之间的新颖协调。该团队将分析NRAO的通用天文软件应用程序(CASA)以及用于天文研究的机器学习(ML)和深度学习(DL)算法在现实数据集上的性能。利用这些结果，他们将设计ML/DL辅助的启发式方法，自适应地确定作业DAG配置和调度，使应用程序能够以最高的性能和最低的系统成本处理地理分布的数据集。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Astronomy has become a data rich science and one of the biggest consumers of computing resources. NSF’s National Radio Astronomy Observatory (NRAO) collects and processes radio telescope data from a broad selection of facilities, but this centralized approach cannot provide timely analysis, driving an increasing need for an efficient astronomical data processing system among distributed observatories. As future telescope projects come online, the data size will grow dramatically and astronomical experiments will require more accurate data analysis and processing, and more detailed results. A critical challenge is job scheduling to maximize application performance and minimize system cost. This project will design and develop system software for geo-distributed observatories, focusing on astronomical data processing performance measurement and performance analysis, and efficient processing system design. Research results will be disseminated through open-source software releases. The project will be showcased in relevant courses and will provide thorough training and collaborative research opportunities to participating undergraduate, graduate, and K-12 students and faculty, and under-represented and female students.This software for scalable and efficient astronomical (SEA) data processing includes novel coordination between job scheduling, job directed acyclic graph (DAG) configuration, and the distribution of data around the globe. The team will analyze the performance of NRAO’s Common Astronomy Software Applications (CASA), and of machine learning (ML) and deep learning (DL) algorithms for astronomical research, on realistic datasets. Using those results, they will design ML/DL-assisted heuristic methods that adaptively determine the job DAG configuration and schedule, allowing an application to process geo-distributed datasets with maximum performance and minimum system cost.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.