NeTS: CSR: Medium: Collaborative Research: Enabling Flexible and High Performance Big Data Analytics Over Geo-Distributed Clouds

大型组织和小型企业都利用遍布地球仪的因特网中心向其用户提供因特网服务。这些站点定期收集与最终用户活动有关的数据以提供更好的服务，并且它们收集服务器监视日志和性能计数器以确保不间断的服务。尽管对这些大型数据集进行快速、高效和经济高效的分析可以显著提高用户体验质量并实现新的应用，但连接数据中心的广域网（WAN）带来了相当大的挑战：由于WAN带宽有限且昂贵，WAN延迟高且可变，分析的性能和及时性都受到广域网的影响。该项目旨在构建一个新的广域网感知的大数据堆栈，用于灵活的地理分布数据分析。该项目不会对可以发出的查询集施加任何约束，并且它将支持各种性能目标，包括获得及时响应，最小化批处理完成时间或使用最小带宽。为了应对WAN条件的不可预测和精细的时间尺度变化，并协调分析堆栈不同层所采取的行动，该项目将实现整体的跨层可见性和优化。它将在堆栈的上层（例如，查询优化）和较低层中的应用级目标（例如，网络）。这将导致对API以及查询优化、查询执行、资源协商、广域存储和网络路由/调度之间的接口进行彻底的重构。该项目的软件工件将被纳入现有的开源大数据堆栈，使研究成果广泛用于公共重用。将提供实验性线束，以确保可重复性并促进后续研究。研究成果将指导行业发展，因为行业慢慢从单一数据中心转向地理分布式设置。该项目有一个实质性的教育部分，涉及在研究生和本科生两级开设关于大数据系统的新课程，其中将涉及使用最先进的大数据软件的实践练习，并将通过大数据靴子训练营向高中生、妇女和代表性不足的少数群体进行宣传。

Large organizations and small enterprises alike leverage datacenters across the globe to offer Internet services to their users. These sites routinely gather data pertaining to end user activities to provide better services, and they collect server monitoring logs and performance counters to ensure uninterrupted service. Although fast, efficient, and cost-effective analyses of these large datasets can significantly improve users' quality of experience and enable novel applications, the wide area network (WAN) that connects the datacenters poses a considerable challenge: because WAN bandwidth is limited and expensive, and WAN latency is high and variable, both the performance and timeliness of analytics are affected by the WAN.This project aims to build a new WAN-aware big data stack customized for flexible geo-distributed data analytics. The project will not impose any constraints on the set of queries that can be issued, and it will support a variety of performance objectives including obtaining timely responses, minimizing batch completion times, or using minimal bandwidth. To account for unpredictable and fine-timescale changes to WAN conditions and to enable coordination among the actions taken by different layers of the analytics stack, this project will enable holistic, cross-layer visibility and optimizations. It will incorporate awareness of the geo-distributed setting in the stack's upper layers (e.g., query optimization) and of application-level objectives in the lower layers (e.g., networking). This will result in a radical re-factoring of the API and interfaces between query optimization, query execution, resource negotiation, wide-area storage, and network routing/scheduling.Software artifacts from this project will be incorporated into existing open source big data stacks, making the research outcomes broadly available for public reuse. The experimental harnesses will be made available to ensure repeatability and to foster follow up research. The research outcomes will guide industry evolution as the industry slowly shifts from single-datacenter to geo-distributed settings. The project has a substantial educational component involving the introduction of new courses on big data systems at both graduate and undergraduate levels that will involve hands-on exercises with state-of-the-art big data software, and it will reach out to high-school students, women, and underrepresented minorities through big data boot camps.