NeTS: CSR: Medium: Collaborative Research: Enabling Flexible and High Performance Big Data Analytics Over Geo-Distributed Clouds

大型组织和小型企业都利用遍布全球的数据中心为其用户提供Internet服务。这些站点定期收集与最终用户活动有关的数据，以提供更好的服务，并收集服务器监视日志和性能计数器，以确保不间断的服务。尽管对这些大型数据集进行快速、高效和经济的分析可以显著提高用户的体验质量并实现新颖的应用，但连接数据中心的广域网（WAN）提出了相当大的挑战：因为广域网带宽有限且昂贵，广域网延迟高且多变，因此分析的性能和及时性都受到广域网的影响。该项目旨在为灵活的地理分布式数据分析定制一个新的wan感知大数据堆栈。该项目不会对可以发出的查询集施加任何限制，并且它将支持各种性能目标，包括获得及时响应、最小化批处理完成时间或使用最小带宽。为了考虑广域网条件的不可预测和精细时间尺度变化，并实现分析堆栈不同层所采取的行动之间的协调，该项目将实现整体的、跨层的可见性和优化。它将在堆栈的上层（例如，查询优化）和低层（例如，网络）中结合对地理分布式设置的感知。这将导致API和查询优化、查询执行、资源协商、广域存储和网络路由/调度之间的接口的彻底重构。该项目的软件构件将被整合到现有的开源大数据栈中，使研究成果广泛地供公众重用。将提供实验装置，以确保可重复性并促进后续研究。随着行业逐渐从单数据中心转向地理分布设置，研究结果将指导行业发展。该项目有大量的教育内容，包括在研究生和本科生阶段引入关于大数据系统的新课程，这些课程将涉及最先进的大数据软件的实践练习，并将通过大数据训练营向高中学生、女性和未被充分代表的少数民族提供服务。

Large organizations and small enterprises alike leverage datacenters across the globe to offer Internet services to their users. These sites routinely gather data pertaining to end user activities to provide better services, and they collect server monitoring logs and performance counters to ensure uninterrupted service. Although fast, efficient, and cost-effective analyses of these large datasets can significantly improve users' quality of experience and enable novel applications, the wide area network (WAN) that connects the datacenters poses a considerable challenge: because WAN bandwidth is limited and expensive, and WAN latency is high and variable, both the performance and timeliness of analytics are affected by the WAN.This project aims to build a new WAN-aware big data stack customized for flexible geo-distributed data analytics. The project will not impose any constraints on the set of queries that can be issued, and it will support a variety of performance objectives including obtaining timely responses, minimizing batch completion times, or using minimal bandwidth. To account for unpredictable and fine-timescale changes to WAN conditions and to enable coordination among the actions taken by different layers of the analytics stack, this project will enable holistic, cross-layer visibility and optimizations. It will incorporate awareness of the geo-distributed setting in the stack's upper layers (e.g., query optimization) and of application-level objectives in the lower layers (e.g., networking). This will result in a radical re-factoring of the API and interfaces between query optimization, query execution, resource negotiation, wide-area storage, and network routing/scheduling.Software artifacts from this project will be incorporated into existing open source big data stacks, making the research outcomes broadly available for public reuse. The experimental harnesses will be made available to ensure repeatability and to foster follow up research. The research outcomes will guide industry evolution as the industry slowly shifts from single-datacenter to geo-distributed settings. The project has a substantial educational component involving the introduction of new courses on big data systems at both graduate and undergraduate levels that will involve hands-on exercises with state-of-the-art big data software, and it will reach out to high-school students, women, and underrepresented minorities through big data boot camps.