Partnership to Advance Throughput Computing (PATh)

合作推进吞吐量计算 (PATH)

• 批准号: 2030508
• 负责人: Miron Livny
• 金额: $ 2250万
• 依托单位: University of Wisconsin-Madison
• 依托单位国家: 美国
• 项目类别: Cooperative Agreement
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2025-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2030508&HistoricalAwards=false
• 关键词: Partnership; Advance; Throughput; Computing; PATh

The Partnership to Advance Throughput Computing (PATh) project will expand Distributed High Throughput Computing (dHTC) technologies and methodologies through innovation, translational effort, and large-scale adoption to advance the Science & Engineering goals of the broader community. The project will offer improved ways for scientists and administrators to manage computing workloads and resources – transforming raw capacity into effective capacity with High Throughput Computing (HTC) principles that maximize the sustained use of distributed computing resources toward computational problems. PATh’s distributed HTC approach will strive to increase the national return on investment of compute resources by enabling institutions to share computing capacity, simultaneously maximizing utilization while also giving smaller campuses easier access to this vital capacity. The techniques advanced by PATh serve the national interest by promoting the progress of science, from academic research groups, to large collaborations, institutions, and industry, with training and outreach components designed and executed to empower the communities that can benefit from these services. PATh is aligned with the NSF National CI Coordination Services blueprint’s goal of producing an “an agile, integrated, robust, trustworthy and sustainable CI ecosystem that drives new thinking and transformative discoveries in all areas of science and engineering (S&E) research and education.” PATh brings together two entities with a strong history of supporting dHTC-enabled research: the Center for High Throughput Computing (CHTC) and the Open Science Grid (OSG) Consortium. The team that founded CHTC at University of Wisconsin–Madison pioneered, in the mid 1990s, the concept and principles of HTC and has advanced and sustained HTCSS ever since. Roughly in parallel and driven by the needs of physics researchers, a diverse and multidisciplinary collaboration laid the foundation of the OSG in the late 1990s to develop, deploy, operate, and sustain a shared and global dHTC ecosystem of sites spanning geographic and administrative boundaries. PATh enables these two partners to align their efforts and leverage each other’s respective strengths, coordinated under the roof of one project. PATh will develop a Fabric of Capacity Services (FoCaS) to build vertical dHTC capabilities and drive the next phase of innovation in the HTCondor Software suite (HTCSS). The HTCSS offers a rich collection of tools for scheduling, federating compute and storage resources, and provisioning these resources to support high-throughput workloads at local, institutional, national, and international scales. In PATh, the HTCSS will receive new capabilities for incorporating external resources (particularly, allocations on HPC machines) into scientific workloads. In addition to the central role of the HTCSS in FoCaS production services, PATh’s core commitment will continue and fortify these services by integrating external technologies and ideas into the dHTC ecosystem. Building on the success of the OSG Connect service, and on campus-oriented Research Computing Facilitation methodologies pioneered by the CHTC at UW-Madison, PATh will create local dHTC expertise and capabilities at a growing number of campuses, including those under the NSF CC* program. Enhanced training, outreach, and community-building activities in PATh will further diversify the reach of dHTC impacts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

合作伙伴关系，以推进高吞吐量计算（PATh）项目将扩大分布式高吞吐量计算（dHTC）技术和方法，通过创新，转化工作，并大规模采用，以推进更广泛的社区的科学工程目标。该项目将为科学家和管理员提供更好的方法来管理计算工作负载和资源-将原始容量转换为有效容量，并采用高吞吐量计算（HTC）原则，最大限度地利用分布式计算资源解决计算问题。PATh的分布式HTC方法将通过使机构能够共享计算能力，同时最大限度地提高利用率，同时也使较小的校园更容易获得这一重要能力，从而努力提高国家对计算资源的投资回报。PATh先进的技术通过促进科学的进步来服务于国家利益，从学术研究团体到大型合作，机构和行业，设计和执行培训和推广组件，以增强可以从这些服务中受益的社区的能力。PATh符合NSF国家CI协调服务蓝图的目标，即建立一个“敏捷，集成，强大，值得信赖和可持续的CI生态系统，推动科学和工程（SE）研究和教育的所有领域的新思维和变革性发现。PATh汇集了两个在支持dHTC支持的研究方面有着悠久历史的实体：高吞吐量计算中心（CHTC）和开放科学网格（OSG）联盟。在威斯康星大学麦迪逊分校创立CHTC的团队在20世纪90年代中期开创了HTC的概念和原则，并从那时起一直在推进和维持HTCSS。在物理研究人员的需求驱动下，一个多样化的多学科合作在20世纪90年代后期奠定了OSG的基础，以开发，部署，运营和维护跨越地理和行政边界的共享和全球dHTC生态系统。PATh使这两个合作伙伴能够在一个项目的框架下协调一致，利用彼此各自的优势。PATh将开发容量服务结构（FoCaS），以构建垂直dHTC功能，并推动HTCondor软件套件（HTCSS）的下一阶段创新。HTCSS提供了一系列丰富的工具，用于调度、联合计算和存储资源，并配置这些资源，以支持本地、机构、国家和国际范围内的高吞吐量工作负载。在PATh中，HTCSS将获得将外部资源（特别是HPC机器上的分配）整合到科学工作负载中的新功能。除了HTCSS在FoCaS生产服务中的核心作用外，PATh的核心承诺将通过将外部技术和想法整合到dHTC生态系统中来继续和加强这些服务。基于OSG Connect服务的成功，以及由CHTC在UW-Madison开创的面向校园的研究计算促进方法，PATh将在越来越多的校园中创建本地dHTC专业知识和能力，包括NSF CC* 计划下的校园。在PATh加强培训，推广和社区建设活动将进一步多样化dHTC影响的范围。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

Analyzing Scientific Data Sharing Patterns for In-network Data Caching

分析网络内数据缓存的科学数据共享模式

• DOI: 10.1145/3452411.3464441
• 发表时间: 2021
• 期刊: SNTA '21: Proceedings of the 2021 on Systems and Network Telemetry and Analytics
• 作者: Copps, Elizabeth;Zhang, Huiyi;Sim, Alex;Wu, Kesheng;Monga, Inder;Guok, Chin;Würthwein, Frank;Davila, Diego;Fajardo, Edgar
• 通讯作者: Fajardo, Edgar

Access Trends of In-network Cache for Scientific Data

科学数据网内缓存访问趋势

• DOI: 10.1145/3526064.3534110
• 发表时间: 2022
• 期刊: SNTA '22: Fifth International Workshop on Systems and Network Telemetry and Analytics
• 作者: Han, Ruize;Sim, Alex;Wu, Kesheng;Monga, Inder;Guok, Chin;Würthwein, Frank;Davila, Diego;Balcas, Justas;Newman, Harvey
• 通讯作者: Newman, Harvey

Analyzing Transatlantic Network Traffic over Scientific Data Caches

通过科学数据缓存分析跨大西洋网络流量

• DOI: 10.1145/3589012.3594897
• 发表时间: 2023
• 期刊: Proceedings of the 2023 on Systems and Network Telemetry and Analytics
• 作者: Deng, Ziyue;Sim, Alex;Wu, Kesheng;Guok, Chin;Hazen, Damian;Monga, Inder;Andrijauskas, Fabio;Würthwein, Frank;Weitzel, Derek
• 通讯作者: Weitzel, Derek

Integrating End-to-End Exascale SDN into the LHC Data Distribution Cyberinfrastructure

将端到端 Exascale SDN 集成到 LHC 数据分发网络基础设施中

• DOI: 10.1145/3491418.3535134
• 发表时间: 2022
• 期刊: Proceedings of PEARC '22: Practice and Experience in Advanced Research Computing
• 作者: Guiang, Jonathan;Arora, Aashay;Davila, Diego;Graham, John;Mishin, Dima;Sfiligoi, Igor;Wuerthwein, Frank;Lehman, Tom;Yang, Xi;Guok, Chin
• 通讯作者: Guok, Chin

The anachronism of whole-GPU accounting

全 GPU 计算的时代错误

• DOI: 10.1145/3491418.3535125
• 发表时间: 2022
• 期刊: Practice and Experience in Advanced Research Computing
• 作者: Sfiligoi, Igor;Schultz, David;Würthwein, Frank;Riedel, Benedikt;Mishin, Dmitry
• 通讯作者: Mishin, Dmitry