Elements: Adaptive End-to-End Parallelism for Distributed Science Workflows

传感和计算技术的技术进步导致科学应用产生的数据量空前增加。随着科学项目在自然界中的分布越来越多，数据大小的增加反过来又导致需要在地理上分布的位置之间移动的流量增加。虽然已经进行了大量投资来建立高速网络，以促进研究和教育机构之间的数据移动，但由于缺乏可扩展的数据传输服务，领域科学家很难有效地利用这种可用的能力。该项目通过开发可扩展和可靠的数据传输服务来满足这一需求。它进一步将数据传输服务集成到弹性工作流管理系统中，以实现分布式科学工作流的端到端优化。该项目对该领域做出了三个新的贡献：（i）它创新了可扩展的完整性验证和文件传输加密，以确保文件传输的可靠性而不牺牲性能。它利用数据传输节点处可用的计算资源来扩展完整性验证和通道加密功能的性能。(ii)它通过将在线传输优化服务集成到弹性工作流管理工具中，为分布式工作流创新了端到端并行性。与现有的工作流管理解决方案，这仅仅是专注于计算任务的优化，在线传输优化服务到弹性工作流调度器的建议集成，使真正的端到端的并行分布式工作流。(iii)最后，它展示了一个真实世界的生物科学工作流程，从NCBI数据库中提取大量的序列读取存档数据流计算就绪SAM/BAM文件的性能开发的服务。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Technological advancements in sensing and computing technologies have led to an unprecedented increase in the amount of data generated by scientific applications. As science projects are increasingly distributed in nature, the increase in data sizes in turn results in an increased volume of traffic that needs to be moved across geographically distributed locations. Although significant investments have been made to build high-speed networks to facilitate data movements between research and education institutions, it is difficult for domain scientists to efficiently utilize this available capacity mainly due to the lack of scalable data transfer services. This project addresses this need by developing a scalable and reliable data transfer service. It further integrates the data transfer service into elastic workflow management systems to achieve end-to-end optimization for distributed science workflows. This project makes three novel contributions to the field: (i) it innovates scalable integrity verification and encryption for file transfers to ensure the reliability of file transfers without sacrificing performance. It takes advantage of computing resources available at data transfer nodes to scale the performance of integrity verification and channel encryption features. (ii) It innovates end-to-end parallelism for distributed workflows by integrating an online transfer optimization service into elastic workflow management tools. Unlike existing workflow management solutions, which merely focus on the optimization of computing tasks, the proposed integration of online transfer optimization services into elastic workflow schedulers enables true end-to-end parallelism for distributed workflows. (iii) Finally, it demonstrates the performance of the developed service on a real-world bioscience workflow that streams a large volume of sequence read archive data from the NCBI database to extract computation-ready SAM/BAM files.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.