Elements: Adaptive End-to-End Parallelism for Distributed Science Workflows

传感和计算技术的技术进步导致科学应用产生的数据量空前增加。由于科学项目在本质上越来越分散，数据大小的增加反过来导致需要在地理分布的位置之间移动的流量增加。尽管已经投入了大量资金来建设高速网络，以促进研究和教育机构之间的数据移动，但由于缺乏可扩展的数据传输服务，领域科学家很难有效地利用这种可用容量。该项目通过开发可伸缩且可靠的数据传输服务来满足这一需求。进一步将数据传输服务集成到弹性工作流管理系统中，实现分布式科学工作流的端到端优化。该项目为该领域做出了三个新颖的贡献：(i)它创新了文件传输的可扩展完整性验证和加密，以确保文件传输的可靠性而不牺牲性能。它利用数据传输节点上可用的计算资源来扩展完整性验证和通道加密特性的性能。（ii）通过将在线传输优化服务集成到弹性工作流管理工具中，创新了分布式工作流的端到端并行性。现有的工作流管理解决方案只关注计算任务的优化，而将在线传输优化服务集成到弹性工作流调度器中，可以实现分布式工作流的真正端到端并行。（iii）最后，它展示了开发的服务在现实世界生物科学工作流中的性能，该工作流从NCBI数据库中流式传输大量序列读取归档数据，以提取计算就绪的SAM/BAM文件。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Technological advancements in sensing and computing technologies have led to an unprecedented increase in the amount of data generated by scientific applications. As science projects are increasingly distributed in nature, the increase in data sizes in turn results in an increased volume of traffic that needs to be moved across geographically distributed locations. Although significant investments have been made to build high-speed networks to facilitate data movements between research and education institutions, it is difficult for domain scientists to efficiently utilize this available capacity mainly due to the lack of scalable data transfer services. This project addresses this need by developing a scalable and reliable data transfer service. It further integrates the data transfer service into elastic workflow management systems to achieve end-to-end optimization for distributed science workflows. This project makes three novel contributions to the field: (i) it innovates scalable integrity verification and encryption for file transfers to ensure the reliability of file transfers without sacrificing performance. It takes advantage of computing resources available at data transfer nodes to scale the performance of integrity verification and channel encryption features. (ii) It innovates end-to-end parallelism for distributed workflows by integrating an online transfer optimization service into elastic workflow management tools. Unlike existing workflow management solutions, which merely focus on the optimization of computing tasks, the proposed integration of online transfer optimization services into elastic workflow schedulers enables true end-to-end parallelism for distributed workflows. (iii) Finally, it demonstrates the performance of the developed service on a real-world bioscience workflow that streams a large volume of sequence read archive data from the NCBI database to extract computation-ready SAM/BAM files.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.