CAREER: Abstractions and Middleware for D3 Science on NSF Distributed Cyberinfrastructure

技术描述：该职业奖项目将开发中间件，以支持分布式网络基础设施（DCI）上的分布式动态数据密集型（D3）科学。现有的nsf资助的CI系统，如极限科学与工程发现环境（XSEDE）和开放科学网格（OSG），使用分布式计算来大幅提高全球研究科学家可用的计算能力；然而，这种分布式系统在处理今天生成的大量数据方面面临着限制。科学仪器和模拟。为了应对这一挑战，PI将开发和部署可扩展的抽象，以促进高性能计算和大规模数据集的集成。基于先前对试点工作的研究，这些新的抽象将实现类似的“试点数据”概念。亲和的连接原理。其结果将是一个统一的概念框架，用于改善数据和计算资源的匹配，并促进动态工作流的放置和调度。通过生成生产级中间件，在一系列DCI系统上完成可扩展的大数据科学，这项研究有可能显著推进科学和工程的多个领域。更广泛的重要性：科学研究人员可用的高性能计算资源越来越多地分布在多个位置的多台机器上。这些资源的集成需要一个中间件结构。在此基础上可以构建和运行各种各样的用户应用程序、工具和服务。然而，随着越来越精确和无处不在的科学仪器和模型产生越来越大的数据量，这种分布式网络基础设施（DCI）面临着前所未有的数据处理挑战，超出了现有DCI中间件的能力。在这个项目中，PI将开发、测试和实施新的中间件解决方案，专门为即将到来的大数据分布式超级计算时代设计。该项目还将编制新的课程、新的教学和推广材料，向中学生和大学生、中学教师和一般公众介绍分布式数据密集型科学这一新兴领域。通过与FutureGrid的合作，PI将设计简单有效的工具，与传统黑人学院和大学（HBCUs）以及其他机构共享这些资源，否则教师开发高级课程材料的机会可能相对有限。该项目还将与道格拉斯科学女性项目（Douglass Program for Women in Science）和罗格斯大学工程与技术女生学院（TARGET）合作，以增加对DCI研究界女学生的参与和支持。

Technical Description: This CAREER Award project will develop middleware to support Distributed Dynamic Data-intensive (D3) science on Distributed Cyberinfrastructure (DCI). Existing NSF-funded CI systems, such as the Extreme Science and Engineering Discovery Environment (XSEDE) and the Open Science Grid (OSG), use distributed computing to substantially increase the computational power available to research scientists around the globe; however, such distributed systems face limitations in their ability to handle the large data-volumes being generated by today?s scientific instruments and simulations. To address this challenge, the PI will develop and deploying extensible abstractions that will facilitate the integration of high-performance computing and large-scale data sets. Building on previous work on pilot-jobs, these new abstractions will implement the analogous concept of ?pilot-data? and the linking principle of ?affinity.? The result will be a unified conceptual framework for improving the matching of data and computing resources and for facilitating dynamic workflow placement and scheduling. This research has the potential to significantly advance multiple areas of science and engineering, by generating production-grade middleware for accomplishing scalable big-data science on a range of DCI systems.Broader Importance: Increasingly, the high-performance computing resources available to scientific researchers are distributed across multiple machines in multiple locations. The integration of these resources requires a fabric of ?middleware,? upon which a wide variety of user applications, tools and services can be built and run. As more accurate and ubiquitous scientific instruments and models produce ever-larger volumes of data, however, this distributed cyberinfrastructure (DCI) is confronting unprecedented data-handling challenges that exceed the capabilities of existing DCI middleware. In this project, the PI will develop, test and implement new middleware solutions, specifically designed for the coming era of big-data distributed supercomputing.The project will also develop new curricula and new teaching and outreach materials for introducing secondary and college students, secondary school teachers, and the general public to the emerging field of distributed data-intensive science. In partnership with FutureGrid, the PI will design simple and effective vehicles for sharing these resources with Historically Black Colleges and Universities (HBCUs) and other institutions where faculty might otherwise have relatively limited opportunity to develop advanced course materials. The PI will also partner with the Douglass Program for Women in Science, and The Academy at Rutgers for Girls in Engineering and Technology (TARGET), to increase engagement of, and support for, female students in the DCI research community.