STCI: Development of Stork Data Scheduler for Mitigating the Data Bottleneck in Petascale Distributed Computing Systems

该提案将使用2009年美国复苏和再投资法案（公法111-5）提供的资金，并符合2009年3月20日白宫备忘录第2节的要求，题为“确保负责任的复苏法案资金支出”。STCI开发用于缓解千万亿级分布式计算系统数据瓶颈的Stork数据调度器项目将增强Stork数据调度器，以缓解千万亿级分布式计算系统中的端到端数据处理瓶颈，并使其作为产品质量软件可用于广泛的用户社区。Stork的新功能包括：数据聚合和缓存；早期错误检测、分类和恢复；集成工作流规划和管理；协议优化；以及数据传输性能预测服务。智力优势：Stork数据调度器将在千万亿级分布式计算领域做出独特的贡献，包括数据放置任务的规划、调度、监控和管理，以及千万亿级分布式应用程序的网络I/O的应用级端到端优化。与现有的方法不同，它将数据资源和与数据访问和移动相关的任务视为第一类实体，就像计算资源和计算任务一样，而不仅仅是计算的副作用。更广泛的影响：该项目不仅会影响传统的科学和工程计算密集型学科，还会影响艺术、人文、商业和教育等新兴计算领域。PI将与分布式数据管理领域的其他领先机构合作，如LBNL， ISI/USC， UNC， UCSD和芝加哥大学/阿贡大学，将Stork与他们的数据管理解决方案集成，并将其传播给他们的用户社区。该项目的综合教育部分将包括科学项目和面向K-12学生（其中99%是少数民族学生）的数据密集型计算暑期训练营，本科生和研究生培训，国际学生/实习生交流计划以及少数民族讲习班。

This proposal will be awarded using funds made available by the American Recovery and Reinvestment Act of 2009 (Public Law 111-5), and meets the requirements established in Section 2 of the White House Memorandum entitled, Ensuring Responsible Spending of Recovery Act Funds, dated March 20, 2009.The STCI Development of Stork Data Scheduler for Mitigating the Data Bottleneck in Petascale Distributed Computing Systems project will enhance the Stork data scheduler to mitigate the end-to-end data handling bottleneck in petascale distributed computing systems and make it available for a wide range of user community as production quality software. New functionalities of Stork will include: data aggregation and caching; early error detection, classification, and recovery; integration with workflow planning and management; optimal protocol tuning; and data transfer performance prediction services.Intellectual Merit:The Stork data scheduler will make a distinctive contribution to petascale distributed computing in the areas of planning, scheduling, monitoring and management of data placement tasks and application-level end-to-end optimization of networked I/O for petascale distributed applications. Unlike existing approaches, it will treat data resources and the tasks related to data access and movement as first class entities just like computational resources and compute tasks, and not simply the side effect of computation. Broader Impact:This project will impact not just traditionally compute intensive disciplines from science and engineering, but also new emerging computational areas in the arts, humanities, business and education. The PI will be collaborating with other leading institutions in the area of distributed data management such as LBNL, ISI/USC, UNC, UCSD, and University Chicago/Argonne to integrate Stork with their data management solutions and disseminate it to their user communities. The comprehensive education component of the project will include science projects and summer training camps on data-intensive computing with K-12 students (where 99% are minority students), undergraduate and graduate student training, international student/intern exchange program, and minority workshops.