SI2-SSE: Scaling up Science on Cyberinfrastructure with the Cooperative Computing Tools

该项目将支持协作计算工具（CCTools）软件的持续开发，该软件使科学家能够将现有的模拟和分析软件代码分发给商业云中的大量计算机、高级超级计算机或两者。 因此，物理学、生物学和化学等领域的研究可以加速，在某些情况下，可以加速一百倍甚至一千倍。 参与该项目的工作人员和学生将开发新的软件功能，并与科学界密切合作，以提高他们在先进计算设施中的生产力。 该项目还将通过培训研究生和本科生的高级软件工程技能来帮助培养技术人才。今天的计算科学家可以使用范围非常广泛的计算设施，包括来自商业云提供商、大学集群或国家计算资源的数千个核心和PB级存储。然而，最终用户通常希望以设计人员没有预料到的方式连接这些系统。他们可能希望在一个设施上访问数据，在另一个设施上运行软件，并在另一个设施上存储结果。然而，组合基础设施比它应该的要困难得多，因为大多数资源假设它们的用户只生活在特定的封闭世界中。在一个环境中构建的程序可能无法在另一个环境中运行;存储在一个系统中的数据可能无法在另一个系统中访问;专门的编程模型在一台机器上可能很快，但在另一台机器上却毫无用处。研究人员不希望被束缚在一个单一的系统;相反，他们希望标准程序可以在所有环境中轻松执行。该项目将创建新功能并支持协作计算工具（CCTools）软件的用户，该软件为科学用户提供利用大规模网络基础设施进行数据密集型科学应用的能力。关键组件是Parrot虚拟文件系统、Makeflow工作流系统和工作队列应用程序框架。这些组件可以单独使用，也可以一起使用，以获取现有的POSIX应用程序，并将其从单个笔记本电脑一直扩展到全国范围的基础设施。该项目将推进CCTools软件的开发，以满足三个关键方面不断变化的技术前景：利用容器技术，有效利用本地并发性，以及在工作流程规模上进行容量管理。这笔赠款将支持一个由学生和工作人员组成的小型团队，他们将参与用户社区，以了解他们的需求，在当前的网络基础设施上维护和扩展软件，并对新老用户进行宣传和教育。该项目将专注于高能物理领域的活跃用户社区，这些社区依赖Parrot在校园集群和开放科学网格中进行全球规模的文件系统访问;生物信息学用户通过VectorBase，LifeMapper和CyVerse学科门户执行复杂的工作流程，以及利用XSEDE和商业云的GPU的合奏分子动力学应用程序。

This project will support the continued development of the Cooperating Computing Tools (CCTools)software, which provides scientists with the ability to distribute existing simulation and analysis software codes to large numbers of computers in commercial clouds, advanced supercomputers, or both. As a result, research in fields such as physics, biology, and chemistry can be accelerated, in some cases, by a hundred- or even a thousand-fold. Staff and students involved in this project will develop new software capabilities and work closely with scientific communities to improve their productivity in advanced computing facilities. The project will also help to develop a technical workforce by training graduate and undergraduate students in advanced software engineering skills.Today's computational scientist has access to an extraordinary range of computing facilities, including thousands of cores and petabytes of storage drawn from commercial cloud providers, university clusters, or national computing resources. However, end users often want to connect these systems in ways that the designers did not anticipate. They may wish to access data at one facility, run software at a second facility, and store the results at yet another. However, combining infrastructure is much harder than it ought to be, because most resources assume that their users only live within that particular closed world. Programs built in one environment may not run in another; data stored in one system may not be accessible in another; specialized programming models may be fast on one machine, but useless on another machine. Researchers do not want to be tied down to a single system; rather, they want standard programs to execute easily across all environments. This project will create new features and support users of the Cooperating Computing Tools (CCTools) software which provides scientific users with the ability to harness large scale cyber-infrastructure for data intensive scientific applications. The key components are the Parrot virtual filesystem, the Makeflow workflow system, and the Work Queue application framework. These components can be used separately or together to take existing POSIX applications and scale them up from a single laptop all the way to national scale infrastructure. This project will advance the development of the CCTools software to meet the changing technology landscape in three key respects: exploiting container technologies, making efficient use of local concurrency, and performing capacity management at the workflow scale. This grant will support a small team of students and staff who will engage user communities to understand their needs, sustain and extend the software on current cyberinfrastructure, and perform outreach and education to users both new and old. The project will focus on active user communities in high energy physics, which rely on Parrot for global scale filesystem access in campus clusters and the Open Science Grid; bioinformatics users executing complex workflows via the VectorBase, LifeMapper, and CyVerse disciplinary portals, and ensemble molecular dynamics applications that harness GPUs from XSEDE and commercial clouds.