DC: Medium: Tackling and Understanding Intermediate Data in Cloud Applications as a First-Class Citizen

DC：中：作为一等公民处理和理解云应用程序中的中间数据

• 批准号: 0964471
• 负责人: Indranil Gupta
• 金额: $ 60万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2010
• 资助国家: 美国
• 起止时间: 2010-07-01 至 2015-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0964471&HistoricalAwards=false
• 关键词: DC; Medium; Tackling; Understanding; Intermediate

Cloud computing infrastructures involve thousands of servers, petabytes of storage, and hundreds of users running various applications that involve gigabytes to terabytes of data. This project focuses on intermediate data that is generated during the execution of parallelized dataflow programs in clouds. Such cloud intermediate data brings forth several unique characteristics: they are massive-scale, distributed, subjected to computational barriers, and prolong job run-times when subjected to server failures. Further, the size of intermediate data in a cloud application is often comparable to or larger than input or output data size, and it can thus range in terabytes. Thus, in spite of extensive existing work on traditional storage problems, there is a critical need for new algorithms and systems that target cloud intermediate data. This project is the first to treat cloud intermediate data as a first-class citizen. The project will involve new algorithm design and analysis, original systems building and implementation, deployment in real world testbeds, and performance of measurement studies. Concretely, this project will build a new system that explicitly manages intermediate data in cloud dataflow programs in order to improve their fault-tolerance, and design and realize barrier relaxation strategies to improve performance of cloud programs. We will implement using open software, deploy, and experimentally evaluate our systems atop the NSF infrastructure called the Cloud Computing Testbed (CCT) that is hosted at the University of Illinois. Finally, we will perform measurement studies of workload characteristics of cloud intermediate data. A fuller understanding of intermediate data in clouds can spawn research in managing cloud infrastructures, improve run-time performance of cloud applications, and lead to new cloud programming paradigms. Our contributions will directly improve the performance and fault-tolerance of applications that are run on the community infrastructure CCT, and positively impact design and deployment of existing and emerging industry clouds. Our results will be published and released in open software and datasets.

云计算基础设施涉及数千台服务器、pb级存储和数百个运行各种应用程序的用户，这些应用程序涉及千兆字节到tb级的数据。这个项目关注的是在云中并行数据流程序执行过程中产生的中间数据。这种云中间数据带来了几个独特的特征：它们是大规模的、分布式的、受到计算障碍的影响，并且在发生服务器故障时延长作业运行时间。此外，云应用程序中中间数据的大小通常与输入或输出数据大小相当或更大，因此可以以tb为单位。因此，尽管在传统存储问题上有大量的现有工作，但迫切需要针对云中间数据的新算法和系统。该项目是第一个将云中间数据视为一等公民的项目。该项目将涉及新的算法设计和分析，原始系统的构建和实现，在现实世界的测试平台中的部署，以及测量研究的性能。具体而言，本项目将构建一个新的系统，明确管理云数据流程序中的中间数据，以提高其容错性，并设计和实现屏障放松策略，以提高云程序的性能。我们将使用开放软件实现、部署和实验评估我们的系统，这些系统位于美国国家科学基金会（NSF）的基础设施之上，该基础设施被称为云计算测试平台（CCT），托管在伊利诺伊大学。最后，我们将对云中间数据的工作负载特征进行测量研究。更全面地理解云中的中间数据可以催生管理云基础设施的研究，提高云应用程序的运行时性能，并产生新的云编程范例。我们的贡献将直接提高在社区基础设施CCT上运行的应用程序的性能和容错性，并对现有和新兴行业云的设计和部署产生积极影响。我们的研究结果将在开放软件和数据集中发表和发布。