Collaborative Research: SHF: Small: Model-driven Design and Optimization of Dataflows for Scientific Applications

高性能计算（HPC）、云计算和边缘计算系统的能力不断增强，直接转化为生成更多数据和执行更多扩展分析的能力，从而扩大了科学家可以在化学、材料科学、分子生物学和药物设计等科学领域使用卷积来研究的自然现象的范围。与此同时，这些数据流复杂性的稳步增长也给将单个数据任务有效组合到可扩展的数据流管道中带来了新的挑战。该项目通过开发解决方案来优化跨异构资源的流水线，从而解决这些关键挑战。该项目建立了一个更广泛的HPC专家社区，他们将对支持科学应用的高性能管道的有效开发产生深远的影响。 研究人员小组通过在Systers（田纳西大学诺克斯维尔的电气工程和计算机科学妇女组织）指导学生，促进代表性不足的学生，特别是妇女的更多参与。此外，研究人员还为早期职业专业人士开发了量身定制的数据分析培训，并与中西部研究计算和数据联盟以及两年一度的NSF/TCPP（并行处理技术社区）并行和分布式计算教育研讨会（EduPar）的与会者分享材料。该项目有四个主要研究组成部分。首先，该项目通过将这些图案映射到真实的科学应用中，定义了科学领域中使用的常见仿射图案的分类，从简单的生产者-消费者对到具有多个生产者和消费者的复杂管道。其次，该项目设计了一个中间件层来处理在HPC、云和边缘资源上执行的并行流水线。第三，该项目开发了一个两步模型，用于缓解管道，这些管道导致数据丢失和低效率，这些数据丢失和低效率与低流量管道中数据生产或消费的放缓有关。最后，该项目培训更广泛的社区利用分类法，中间件和模型，通过识别潜在的瓶颈并进行必要的调整来优化真实的科学应用，以最大限度地提高管道效率和准确性，持续监控和优化管道，以确保最高质量的科学产出。该奖项反映了NSF的法定使命，并通过使用基金会的学术价值和更广泛的影响评审标准。

The increasing capability of high-performance computing (HPC), cloud computing, and edge computing systems directly translates into the ability to generate more data and execute more extended analyses, thus expanding the range of natural phenomena that scientists can study using dataflows in scientific domains such as chemistry, materials sciences, molecular biology, and drug design. At the same time, the steady growth in the complexity of these dataflows also results in new challenges in the effective composition of single data tasks into scalable dataflow pipelines. This project addresses these critical challenges by developing solutions to optimize dataflow pipelines across heterogeneous resources. This project builds a broader community of HPC experts, who will have a far-reaching impact on the efficient development of dataflow pipelines supporting scientific applications. The team of researchers promotes increased participation of underrepresented students, particularly women, through mentoring students in Systers (the organization for women in Electrical Engineering and Computer Science at the University of Tennessee Knoxville). Furthermore, the researchers develop data analytics training tailored for early career professionals and share the material with the Midwest Research Computing and Data Consortium and the attendees at the bi-annual NSF/TCPP (Technical Community on Parallel Processing) workshops on parallel and distributed computing education (EduPar). This project has four main research components. First, the project defines a taxonomy of common dataflow motifs used in scientific domains, ranging from simple producer-consumer pairs to complex pipelines with multiple producers and consumers, by mapping these motifs to real scientific applications. Second, the project designs a middleware layer to handle dataflow pipelines executing on HPC, cloud, and edge resources. Third, the project develops a 2-step model for mitigating pipelines that result in data loss and inefficiencies associated with the slowdown in data production or consumption in dataflow pipelines. Finally, the project trains a broader community to utilize the taxonomy, middleware, and model to optimize real scientific applications by identifying potential bottlenecks and making necessary adjustments to maximize pipeline efficiency and accuracy, continuously monitoring and optimizing pipelines to ensure the highest quality scientific output possible.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.