NeTS: CSR: Medium: Collaborative Research: Enabling Flexible and High Performance Big Data Analytics Over Geo-Distributed Clouds

大型组织和小型企业都利用全球各地的数据中心向其用户提供互联网服务。这些站点定期收集与终端用户活动有关的数据以提供更好的服务，并收集服务器监控日志和性能计数器以确保服务不中断。虽然对这些大型数据集的快速、高效和经济高效的分析可以显著提高用户的体验质量，并支持新的应用程序，但连接数据中心的广域网(WAN)构成了一个相当大的挑战：由于广域网带宽有限且昂贵，并且广域网延迟高且可变，分析的性能和及时性都会受到广域网的影响。该项目旨在构建一个新的广域网感知的大数据堆栈，用于灵活的地理分布数据分析。该项目不会对可以发出的查询集施加任何限制，它将支持各种性能目标，包括获得及时响应、最大限度地减少批量完成时间或使用最小带宽。为了应对广域网状况的不可预测和精细时间尺度变化，并实现分析堆栈不同层采取的操作之间的协调，该项目将实现全面、跨层的可见性和优化。它将包括对堆栈上层的地理分布设置(例如，查询优化)和较低层的应用级目标(例如，联网)的认识。这将导致API和查询优化、查询执行、资源协商、广域存储和网络路由/调度之间的接口的彻底重构。该项目的软件构件将被整合到现有的开源大数据堆栈中，使研究成果广泛可供公众重复使用。将提供实验装置，以确保重复性和促进后续研究。随着行业从单数据中心向地理分布式环境的缓慢转变，研究成果将指导行业演变。该项目有一个重要的教育部分，包括在研究生和本科生层面引入关于大数据系统的新课程，其中将包括使用最先进的大数据软件的动手练习，并将通过大数据新兵训练营接触到高中生、女性和代表性不足的少数族裔。

Large organizations and small enterprises alike leverage datacenters across the globe to offer Internet services to their users. These sites routinely gather data pertaining to end user activities to provide better services, and they collect server monitoring logs and performance counters to ensure uninterrupted service. Although fast, efficient, and cost-effective analyses of these large datasets can significantly improve users' quality of experience and enable novel applications, the wide area network (WAN) that connects the datacenters poses a considerable challenge: because WAN bandwidth is limited and expensive, and WAN latency is high and variable, both the performance and timeliness of analytics are affected by the WAN.This project aims to build a new WAN-aware big data stack customized for flexible geo-distributed data analytics. The project will not impose any constraints on the set of queries that can be issued, and it will support a variety of performance objectives including obtaining timely responses, minimizing batch completion times, or using minimal bandwidth. To account for unpredictable and fine-timescale changes to WAN conditions and to enable coordination among the actions taken by different layers of the analytics stack, this project will enable holistic, cross-layer visibility and optimizations. It will incorporate awareness of the geo-distributed setting in the stack's upper layers (e.g., query optimization) and of application-level objectives in the lower layers (e.g., networking). This will result in a radical re-factoring of the API and interfaces between query optimization, query execution, resource negotiation, wide-area storage, and network routing/scheduling.Software artifacts from this project will be incorporated into existing open source big data stacks, making the research outcomes broadly available for public reuse. The experimental harnesses will be made available to ensure repeatability and to foster follow up research. The research outcomes will guide industry evolution as the industry slowly shifts from single-datacenter to geo-distributed settings. The project has a substantial educational component involving the introduction of new courses on big data systems at both graduate and undergraduate levels that will involve hands-on exercises with state-of-the-art big data software, and it will reach out to high-school students, women, and underrepresented minorities through big data boot camps.