Collaborative Research: SHF: Small: Model-driven Design and Optimization of Dataflows for Scientific Applications

高性能计算(HPC)、云计算和边缘计算系统的能力不断增强，直接转化为生成更多数据和执行更广泛分析的能力，从而扩大了科学家可以利用化学、材料科学、分子生物学和药物设计等科学领域的数据流研究的自然现象的范围。与此同时，这些数据流的复杂性的稳步增长也导致了将单个数据任务有效地组合到可扩展的数据流管道中的新挑战。该项目通过开发解决方案来优化跨不同资源的数据流管道，从而解决了这些关键挑战。该项目建立了一个更广泛的高性能计算专家社区，他们将对支持科学应用的数据流管道的高效开发产生深远影响。研究小组通过在Syster(田纳西诺克斯维尔大学电气工程和计算机科学女性组织)指导学生，促进了代表不足的学生，特别是女性的更多参与。此外，研究人员还为职业生涯早期的专业人员开发数据分析培训，并与中西部研究计算和数据联盟以及每两年举办一次的NSF/TCPP(并行处理技术社区)并行和分布式计算教育(EduPar)研讨会的与会者分享这些材料。该项目有四个主要研究部分。首先，该项目定义了科学领域中使用的常见数据流主题的分类，从简单的生产者-消费者对到具有多个生产者和消费者的复杂管道，通过将这些主题映射到真正的科学应用。其次，项目设计了一个中间件层来处理在HPC、云和边缘资源上执行的数据流管道。第三，该项目开发了一个两步模型，用于缓解导致数据丢失和与数据流管道中数据生产或消费放缓相关的低效率的管道。最后，该项目培训更广泛的社区利用分类法、中间件和模型来优化真正的科学应用，方法是识别潜在的瓶颈并进行必要的调整，以最大限度地提高管道的效率和准确性，持续监控和优化管道，以确保尽可能高质量的科学产出。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The increasing capability of high-performance computing (HPC), cloud computing, and edge computing systems directly translates into the ability to generate more data and execute more extended analyses, thus expanding the range of natural phenomena that scientists can study using dataflows in scientific domains such as chemistry, materials sciences, molecular biology, and drug design. At the same time, the steady growth in the complexity of these dataflows also results in new challenges in the effective composition of single data tasks into scalable dataflow pipelines. This project addresses these critical challenges by developing solutions to optimize dataflow pipelines across heterogeneous resources. This project builds a broader community of HPC experts, who will have a far-reaching impact on the efficient development of dataflow pipelines supporting scientific applications. The team of researchers promotes increased participation of underrepresented students, particularly women, through mentoring students in Systers (the organization for women in Electrical Engineering and Computer Science at the University of Tennessee Knoxville). Furthermore, the researchers develop data analytics training tailored for early career professionals and share the material with the Midwest Research Computing and Data Consortium and the attendees at the bi-annual NSF/TCPP (Technical Community on Parallel Processing) workshops on parallel and distributed computing education (EduPar). This project has four main research components. First, the project defines a taxonomy of common dataflow motifs used in scientific domains, ranging from simple producer-consumer pairs to complex pipelines with multiple producers and consumers, by mapping these motifs to real scientific applications. Second, the project designs a middleware layer to handle dataflow pipelines executing on HPC, cloud, and edge resources. Third, the project develops a 2-step model for mitigating pipelines that result in data loss and inefficiencies associated with the slowdown in data production or consumption in dataflow pipelines. Finally, the project trains a broader community to utilize the taxonomy, middleware, and model to optimize real scientific applications by identifying potential bottlenecks and making necessary adjustments to maximize pipeline efficiency and accuracy, continuously monitoring and optimizing pipelines to ensure the highest quality scientific output possible.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.