SI2-SSE: Scaling up Science on Cyberinfrastructure with the Cooperative Computing Tools

该项目将支持合作计算工具(CCTools)软件的继续开发，该软件使科学家能够将现有的模拟和分析软件代码分发到商业云中的大量计算机或高级超级计算机，或两者兼而有之。因此，物理、生物和化学等领域的研究可以加速，在某些情况下，可以加快100倍甚至1000倍。参与该项目的工作人员和学生将开发新的软件能力，并与科学界密切合作，以提高他们在先进计算设施方面的生产率。该项目还将通过培训研究生和本科生的高级软件工程技能来帮助培养一支技术队伍。今天的计算科学家可以使用非常广泛的计算设施，包括来自商业云提供商、大学集群或国家计算资源的数千个核心和PB的存储。然而，最终用户通常希望以设计者没有预料到的方式连接这些系统。他们可能希望访问一个设施的数据，在第二个设施运行软件，并将结果存储在另一个设施。然而，组合基础设施比应有的要难得多，因为大多数资源假设其用户只生活在那个特定的封闭世界中。在一个环境中构建的程序可能无法在另一个环境中运行；存储在一个系统中的数据可能无法在另一个系统中访问；专用编程模型在一台计算机上可能很快，但在另一台计算机上却毫无用处。研究人员不想被束缚在一个单一的系统上；相反，他们希望标准程序能够在所有环境中轻松执行。该项目将创建新功能并支持合作计算工具(CCTools)软件的用户，该软件为科学用户提供利用大规模网络基础设施进行数据密集型科学应用的能力。关键组件是Parrot虚拟文件系统、Makeflow工作流系统和工作队列应用程序框架。这些组件可以单独使用，也可以一起使用，以采用现有的POSIX应用程序，并将它们从一台笔记本电脑一直扩展到全国性的基础设施。该项目将在三个关键方面推进CCTools软件的开发，以满足不断变化的技术格局：开发集装箱技术、有效利用本地并发以及在工作流规模上执行容量管理。这笔赠款将支持一个由学生和工作人员组成的小组，他们将使用户社区了解他们的需求，在现有的网络基础设施上维持和扩展该软件，并对新老用户进行外联和教育。该项目将专注于高能物理领域的活跃用户社区，这些社区依赖Parot在校园集群和开放科学网格中进行全球范围的文件系统访问；生物信息学用户通过VectorBase、LifeMapper和CyVerse学科门户网站执行复杂的工作流程，以及利用XSEDE和商业云中的GPU的分子动力学应用程序。

This project will support the continued development of the Cooperating Computing Tools (CCTools)software, which provides scientists with the ability to distribute existing simulation and analysis software codes to large numbers of computers in commercial clouds, advanced supercomputers, or both. As a result, research in fields such as physics, biology, and chemistry can be accelerated, in some cases, by a hundred- or even a thousand-fold. Staff and students involved in this project will develop new software capabilities and work closely with scientific communities to improve their productivity in advanced computing facilities. The project will also help to develop a technical workforce by training graduate and undergraduate students in advanced software engineering skills.Today's computational scientist has access to an extraordinary range of computing facilities, including thousands of cores and petabytes of storage drawn from commercial cloud providers, university clusters, or national computing resources. However, end users often want to connect these systems in ways that the designers did not anticipate. They may wish to access data at one facility, run software at a second facility, and store the results at yet another. However, combining infrastructure is much harder than it ought to be, because most resources assume that their users only live within that particular closed world. Programs built in one environment may not run in another; data stored in one system may not be accessible in another; specialized programming models may be fast on one machine, but useless on another machine. Researchers do not want to be tied down to a single system; rather, they want standard programs to execute easily across all environments. This project will create new features and support users of the Cooperating Computing Tools (CCTools) software which provides scientific users with the ability to harness large scale cyber-infrastructure for data intensive scientific applications. The key components are the Parrot virtual filesystem, the Makeflow workflow system, and the Work Queue application framework. These components can be used separately or together to take existing POSIX applications and scale them up from a single laptop all the way to national scale infrastructure. This project will advance the development of the CCTools software to meet the changing technology landscape in three key respects: exploiting container technologies, making efficient use of local concurrency, and performing capacity management at the workflow scale. This grant will support a small team of students and staff who will engage user communities to understand their needs, sustain and extend the software on current cyberinfrastructure, and perform outreach and education to users both new and old. The project will focus on active user communities in high energy physics, which rely on Parrot for global scale filesystem access in campus clusters and the Open Science Grid; bioinformatics users executing complex workflows via the VectorBase, LifeMapper, and CyVerse disciplinary portals, and ensemble molecular dynamics applications that harness GPUs from XSEDE and commercial clouds.