Elements: Adaptive End-to-End Parallelism for Distributed Science Workflows

传感和计算技术的进步导致科学应用产生的数据量出现了前所未有的增长。随着科学项目在本质上越来越分散，数据大小的增加反过来又导致需要在地理上分布的位置之间移动的通信量增加。尽管已经进行了大量投资来建设高速网络以促进研究和教育机构之间的数据移动，但领域科学家很难有效地利用这种可用容量，主要是因为缺乏可扩展的数据传输服务。该项目通过开发可扩展和可靠的数据传输服务来满足这一需求。它进一步将数据传输服务集成到弹性工作流管理系统中，实现了分布式科学工作流的端到端优化。该项目在三个方面做出了新的贡献：(I)创新了可扩展的文件传输完整性验证和加密，在不牺牲性能的情况下确保文件传输的可靠性。它利用数据传输节点上可用的计算资源来扩展完整性验证和通道加密功能的性能。(2)通过将在线传输优化服务集成到弹性工作流管理工具中，创新了分布式工作流的端到端并行性。与只关注计算任务优化的现有工作流管理解决方案不同，建议将在线传输优化服务集成到弹性工作流调度器中，从而为分布式工作流实现真正的端到端并行。(Iii)最后，它展示了开发的服务在真实世界生物科学工作流程中的表现，该工作流程从NCBI数据库中传输大量序列读取档案数据，以提取可用于计算的SAM/BAM文件。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Technological advancements in sensing and computing technologies have led to an unprecedented increase in the amount of data generated by scientific applications. As science projects are increasingly distributed in nature, the increase in data sizes in turn results in an increased volume of traffic that needs to be moved across geographically distributed locations. Although significant investments have been made to build high-speed networks to facilitate data movements between research and education institutions, it is difficult for domain scientists to efficiently utilize this available capacity mainly due to the lack of scalable data transfer services. This project addresses this need by developing a scalable and reliable data transfer service. It further integrates the data transfer service into elastic workflow management systems to achieve end-to-end optimization for distributed science workflows. This project makes three novel contributions to the field: (i) it innovates scalable integrity verification and encryption for file transfers to ensure the reliability of file transfers without sacrificing performance. It takes advantage of computing resources available at data transfer nodes to scale the performance of integrity verification and channel encryption features. (ii) It innovates end-to-end parallelism for distributed workflows by integrating an online transfer optimization service into elastic workflow management tools. Unlike existing workflow management solutions, which merely focus on the optimization of computing tasks, the proposed integration of online transfer optimization services into elastic workflow schedulers enables true end-to-end parallelism for distributed workflows. (iii) Finally, it demonstrates the performance of the developed service on a real-world bioscience workflow that streams a large volume of sequence read archive data from the NCBI database to extract computation-ready SAM/BAM files.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.