SI2-SSE: Scaling up Science on Cyberinfrastructure with the Cooperative Computing Tools

SI2-SSE：利用协作计算工具扩大网络基础设施科学规模

• 批准号: 1642409
• 负责人: Douglas Thain
• 金额: $ 49.47万
• 依托单位: University of Notre Dame
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-09-01 至 2020-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1642409&HistoricalAwards=false
• 关键词: SI2; SSE; Scaling; up; Science

This project will support the continued development of the Cooperating Computing Tools (CCTools)software, which provides scientists with the ability to distribute existing simulation and analysis software codes to large numbers of computers in commercial clouds, advanced supercomputers, or both. As a result, research in fields such as physics, biology, and chemistry can be accelerated, in some cases, by a hundred- or even a thousand-fold. Staff and students involved in this project will develop new software capabilities and work closely with scientific communities to improve their productivity in advanced computing facilities. The project will also help to develop a technical workforce by training graduate and undergraduate students in advanced software engineering skills.Today's computational scientist has access to an extraordinary range of computing facilities, including thousands of cores and petabytes of storage drawn from commercial cloud providers, university clusters, or national computing resources. However, end users often want to connect these systems in ways that the designers did not anticipate. They may wish to access data at one facility, run software at a second facility, and store the results at yet another. However, combining infrastructure is much harder than it ought to be, because most resources assume that their users only live within that particular closed world. Programs built in one environment may not run in another; data stored in one system may not be accessible in another; specialized programming models may be fast on one machine, but useless on another machine. Researchers do not want to be tied down to a single system; rather, they want standard programs to execute easily across all environments. This project will create new features and support users of the Cooperating Computing Tools (CCTools) software which provides scientific users with the ability to harness large scale cyber-infrastructure for data intensive scientific applications. The key components are the Parrot virtual filesystem, the Makeflow workflow system, and the Work Queue application framework. These components can be used separately or together to take existing POSIX applications and scale them up from a single laptop all the way to national scale infrastructure. This project will advance the development of the CCTools software to meet the changing technology landscape in three key respects: exploiting container technologies, making efficient use of local concurrency, and performing capacity management at the workflow scale. This grant will support a small team of students and staff who will engage user communities to understand their needs, sustain and extend the software on current cyberinfrastructure, and perform outreach and education to users both new and old. The project will focus on active user communities in high energy physics, which rely on Parrot for global scale filesystem access in campus clusters and the Open Science Grid; bioinformatics users executing complex workflows via the VectorBase, LifeMapper, and CyVerse disciplinary portals, and ensemble molecular dynamics applications that harness GPUs from XSEDE and commercial clouds.

该项目将支持合作计算工具(CCTools)软件的继续开发，该软件使科学家能够将现有的模拟和分析软件代码分发到商业云中的大量计算机或高级超级计算机，或两者兼而有之。因此，物理、生物和化学等领域的研究可以加速，在某些情况下，可以加快100倍甚至1000倍。参与该项目的工作人员和学生将开发新的软件能力，并与科学界密切合作，以提高他们在先进计算设施方面的生产率。该项目还将通过培训研究生和本科生的高级软件工程技能来帮助培养一支技术队伍。今天的计算科学家可以使用非常广泛的计算设施，包括来自商业云提供商、大学集群或国家计算资源的数千个核心和PB的存储。然而，最终用户通常希望以设计者没有预料到的方式连接这些系统。他们可能希望访问一个设施的数据，在第二个设施运行软件，并将结果存储在另一个设施。然而，组合基础设施比应有的要难得多，因为大多数资源假设其用户只生活在那个特定的封闭世界中。在一个环境中构建的程序可能无法在另一个环境中运行；存储在一个系统中的数据可能无法在另一个系统中访问；专用编程模型在一台计算机上可能很快，但在另一台计算机上却毫无用处。研究人员不想被束缚在一个单一的系统上；相反，他们希望标准程序能够在所有环境中轻松执行。该项目将创建新功能并支持合作计算工具(CCTools)软件的用户，该软件为科学用户提供利用大规模网络基础设施进行数据密集型科学应用的能力。关键组件是Parrot虚拟文件系统、Makeflow工作流系统和工作队列应用程序框架。这些组件可以单独使用，也可以一起使用，以采用现有的POSIX应用程序，并将它们从一台笔记本电脑一直扩展到全国性的基础设施。该项目将在三个关键方面推进CCTools软件的开发，以满足不断变化的技术格局：开发集装箱技术、有效利用本地并发以及在工作流规模上执行容量管理。这笔赠款将支持一个由学生和工作人员组成的小组，他们将使用户社区了解他们的需求，在现有的网络基础设施上维持和扩展该软件，并对新老用户进行外联和教育。该项目将专注于高能物理领域的活跃用户社区，这些社区依赖Parot在校园集群和开放科学网格中进行全球范围的文件系统访问；生物信息学用户通过VectorBase、LifeMapper和CyVerse学科门户网站执行复杂的工作流程，以及利用XSEDE和商业云中的GPU的分子动力学应用程序。

项目成果

SHADHO: Massively Scalable Hardware-Aware Distributed Hyperparameter Optimization

• DOI: 10.1109/wacv.2018.00086
• 发表时间: 2017-07
• 期刊: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV)
• 作者: Jeffery Kinnison;Nathaniel Kremer-Herman;D. Thain;W. Scheirer

Log Discovery for Troubleshooting Open Distributed Systems with TLQ

• DOI: 10.1145/3311790.3396633
• 发表时间: 2020-07
• 期刊: Practice and Experience in Advanced Research Computing
• 作者: Nathaniel Kremer-Herman;D. Thain

Conducting reproducible research with Umbrella: Tracking, creating, and preserving execution environments

使用 Umbrella 进行可重复的研究：跟踪、创建和保留执行环境

• DOI: 10.1109/escience.2016.7870889
• 发表时间: 2016
• 期刊: IEEE Conference on e-Science
• 作者: Meng, Haiyan;Thain, Douglas;Vyushkov, Alexander;Wolf, Matthias;Woodard, Anna
• 通讯作者: Woodard, Anna

Opportunistic Computing with Lobster: Lessons Learned from Scaling up to 25k Non-Dedicated Cores

使用 Lobster 进行机会计算：从扩展到 25k 个非专用核心中吸取的经验教训

• DOI: 10.1088/1742-6596/898/5/052036
• 发表时间: 2017
• 期刊: Journal of Physics: Conference Series
• 作者: Wolf, Matthias;Woodard, Anna;Li, Wenzhao;Anampa, Kenyi Hurtado;Yannakopoulos, Anna;Tovar, Benjamin;Donnelly, Patrick;Brenner, Paul;Lannon, Kevin;Hildreth, Mike
• 通讯作者: Hildreth, Mike

Dynamic Sizing of Continuously Divisible Jobs for Heterogeneous Resources

异构资源连续可分作业的动态调整

• DOI: 10.1109/escience.2019.00026
• 发表时间: 2019
• 期刊: IEEE International Conference on e-Science
• 作者: Hazekamp, Nicholas;Tovar, Benjamin;Thain, Douglas
• 通讯作者: Thain, Douglas