STCI: Development of Stork Data Scheduler for Mitigating the Data Bottleneck in Petascale Distributed Computing Systems

该提案将使用2009年美国复苏和再投资法案提供的资金进行奖励（公法111-5），并符合3月20日名为“确保负责任地支出复苏法案资金”的白宫备忘录第2节规定的要求，2009年，STCI开发Stork数据调度器以缓解Petascale分布式计算系统中的数据瓶颈项目将增强Stork数据调度器，以缓解端到端的数据瓶颈。结束千万亿次分布式计算系统中的数据处理瓶颈，并使其作为产品质量软件可供广泛的用户社区使用。Stork的新功能将包括：数据聚合和缓存;早期错误检测、分类和恢复;与工作流规划和管理的集成;最佳协议调整;以及数据传输性能预测服务。智力优势：Stork数据调度器将在规划、调度、监控和管理数据放置任务以及应用级端到端优化网络I/O等领域为千万亿次分布式应用程序做出独特的贡献。与现有方法不同，它将数据资源和与数据访问和移动相关的任务视为第一类实体，就像计算资源和计算任务一样，而不仅仅是计算的副作用。更广泛的影响：该项目不仅将影响科学和工程领域的传统计算密集型学科，还将影响艺术、人文、商业和教育领域的新兴计算领域。PI将与分布式数据管理领域的其他领先机构合作，如LBNL，ISI/USC，UCSD和芝加哥大学/阿贡，将Stork与他们的数据管理解决方案集成，并将其传播到他们的用户社区。该项目的综合教育部分将包括科学项目和K-12学生（其中99%是少数民族学生）的数据密集型计算夏令营，本科生和研究生培训，国际学生/实习生交流计划和少数民族研讨会。

This proposal will be awarded using funds made available by the American Recovery and Reinvestment Act of 2009 (Public Law 111-5), and meets the requirements established in Section 2 of the White House Memorandum entitled, Ensuring Responsible Spending of Recovery Act Funds, dated March 20, 2009.The STCI Development of Stork Data Scheduler for Mitigating the Data Bottleneck in Petascale Distributed Computing Systems project will enhance the Stork data scheduler to mitigate the end-to-end data handling bottleneck in petascale distributed computing systems and make it available for a wide range of user community as production quality software. New functionalities of Stork will include: data aggregation and caching; early error detection, classification, and recovery; integration with workflow planning and management; optimal protocol tuning; and data transfer performance prediction services.Intellectual Merit:The Stork data scheduler will make a distinctive contribution to petascale distributed computing in the areas of planning, scheduling, monitoring and management of data placement tasks and application-level end-to-end optimization of networked I/O for petascale distributed applications. Unlike existing approaches, it will treat data resources and the tasks related to data access and movement as first class entities just like computational resources and compute tasks, and not simply the side effect of computation. Broader Impact:This project will impact not just traditionally compute intensive disciplines from science and engineering, but also new emerging computational areas in the arts, humanities, business and education. The PI will be collaborating with other leading institutions in the area of distributed data management such as LBNL, ISI/USC, UNC, UCSD, and University Chicago/Argonne to integrate Stork with their data management solutions and disseminate it to their user communities. The comprehensive education component of the project will include science projects and summer training camps on data-intensive computing with K-12 students (where 99% are minority students), undergraduate and graduate student training, international student/intern exchange program, and minority workshops.