DC: Medium: Tackling and Understanding Intermediate Data in Cloud Applications as a First-Class Citizen

云计算基础设施涉及数千台服务器、PB级存储以及数百名运行各种应用程序的用户，这些应用程序涉及千兆字节到兆兆字节的数据。这个项目的重点是在云中执行并行化的程序期间生成的中间数据。这样的云中间数据带来了几个独特的特征：它们是大规模的，分布式的，受到计算障碍的影响，并且当受到服务器故障的影响时会延长作业运行时间。此外，云应用程序中的中间数据的大小通常与输入或输出数据大小相当或更大，因此其范围可以是TB。因此，尽管对传统存储问题进行了大量的现有工作，但仍然迫切需要针对云中间数据的新算法和系统。该项目是第一个将云中间数据视为一等公民的项目。该项目将涉及新的算法设计和分析，原始系统的构建和实施，部署在真实的世界的测试平台，和性能的测量研究。具体而言，本项目将建立一个新的系统，显式地管理云计算程序中的中间数据，以提高其容错性，并设计和实现障碍松弛策略，以提高云程序的性能。我们将使用开放软件实现，部署和实验评估我们的系统在NSF基础设施之上，称为云计算测试床（CCT），托管在伊利诺伊大学。最后，我们将对云中间数据的工作负载特性进行测量研究。 对云中中间数据的更全面理解可以促进管理云基础设施的研究，提高云应用程序的运行时性能，并导致新的云编程范式。我们的贡献将直接提高在社区基础设施CCT上运行的应用程序的性能和容错能力，并对现有和新兴行业云的设计和部署产生积极影响。我们的研究结果将在开放软件和数据集中发布。

Cloud computing infrastructures involve thousands of servers, petabytes of storage, and hundreds of users running various applications that involve gigabytes to terabytes of data. This project focuses on intermediate data that is generated during the execution of parallelized dataflow programs in clouds. Such cloud intermediate data brings forth several unique characteristics: they are massive-scale, distributed, subjected to computational barriers, and prolong job run-times when subjected to server failures. Further, the size of intermediate data in a cloud application is often comparable to or larger than input or output data size, and it can thus range in terabytes. Thus, in spite of extensive existing work on traditional storage problems, there is a critical need for new algorithms and systems that target cloud intermediate data. This project is the first to treat cloud intermediate data as a first-class citizen. The project will involve new algorithm design and analysis, original systems building and implementation, deployment in real world testbeds, and performance of measurement studies. Concretely, this project will build a new system that explicitly manages intermediate data in cloud dataflow programs in order to improve their fault-tolerance, and design and realize barrier relaxation strategies to improve performance of cloud programs. We will implement using open software, deploy, and experimentally evaluate our systems atop the NSF infrastructure called the Cloud Computing Testbed (CCT) that is hosted at the University of Illinois. Finally, we will perform measurement studies of workload characteristics of cloud intermediate data. A fuller understanding of intermediate data in clouds can spawn research in managing cloud infrastructures, improve run-time performance of cloud applications, and lead to new cloud programming paradigms. Our contributions will directly improve the performance and fault-tolerance of applications that are run on the community infrastructure CCT, and positively impact design and deployment of existing and emerging industry clouds. Our results will be published and released in open software and datasets.