CSR: Small: Highly Efficient, Pipeline-oriented Data-intensive Scalable Computing

智力优势：数据密集型可扩展计算（DISC）对于从搜索引擎和社交网络到生物和科学应用的千万亿次规模问题越来越重要。 为支持大规模DISC计算而构建的数据中心已经以惊人的规模运行，容纳了数十万个计算节点、EB级存储和PB级内存。 当前的DISC系统已经通过可伸缩性解决了这些数据大小，但是由此产生的每节点性能落后于每服务器容量超过一个数量级。 例如，在当前系统中，多达94%的可用磁盘I/O和33%的CPU保持空闲。 这导致不可持续的成本和能源需求。 满足未来的数据处理的挑战将只有可能，如果DISC系统可以部署在一个可持续的，有效的方式。这个项目的重点是两个具体的，未解决的挑战，建设和部署可持续的DISC系统： - 随着系统的扩展，缺乏每个节点的效率和跨资源平衡，以及 - 针对DISC工作负载量身定制的高效存储容错。 该项目的方法是在系统设计期间静态地以及在运行时期间动态地自动和动态地确保计算、内存、网络和底层存储组件之间的跨资源平衡。 目标是以平衡的方式支持一般DISC处理，而不管应用程序行为和异构计算和存储配置的变化。 这项工作将产生一个功能齐全的原型DISC系统，支持Map/Reduce编程模型，以支持通用应用程序。 - 培训不同的学生，如本科生和代表性不足的群体-了解DISC服务作为整体课程的一个有趣的部分，并作为跨学科合作的资源。 - 建议的平衡运行时系统的公开发布，包括对更高级别编程模型的支持; - 与工业合作伙伴合作，作为UCSD网络系统中心的一部分，以解决工业和政府数据处理这一关键部分的可持续性和效率问题。

Intellectual Merit: Data-intensive Scalable Computing (DISC) is increasingly important to peta-scale problems, from search engines and social networks to biological and scientific applications. Already datacenters built to support large-scale DISC computing operate at staggering scale, housing up to hundreds of thousands of compute nodes, exabytes of storage, and petabytes of memory. Current DISC systems have addressed these data sizes through scalability, however the resulting per-node performance has lagged behind per-server capacity by more than an order of magnitude. For example, in current systems as much as 94% of available disk I/O and 33% of CPU remain idle. This results in unsustainable cost and energy requirements. Meeting future data processing challenges will only be possible if DISC systems can be deployed in a sustainable, efficient manner.This project focuses on two specific, unaddressed challenges to building and deploying sustainable DISC systems: -a lack of per-node efficiency and cross-resource balance as the system scales, and -highly-efficient storage fault tolerance tailored to DISC workloads. This project's approach is to automatically and dynamically ensure cross-resource balance between compute, memory, network, and underlying storage components statically during system design, as well as dynamically during runtime. The goal is to support general DISC processing in a balanced manner despite changing application behavior and heterogeneous computing and storage configurations. This work will result in a fully functional prototype DISC system supporting the Map/Reduce programming model to support general-purpose application programs.Broader impacts include: -training diverse students, such as undergraduates and underrepresented groups - to understand DISC services as an interesting part of the overall curriculum and as a resource for interdisciplinary collaboration. -a public release of the proposed balanced runtime system, including support for higher-level programming models; -working with industrial partners as part of UCSD's Center for Networked Systems to address sustainability and efficiency issues in this critical portion of industrial and governmental data processing.