Managing Resources across Geo-Distributed Datacenters for Big Data Analytics and Mobile Cloud Applications

跨地理分布式数据中心管理资源以进行大数据分析和移动云应用程序

• 批准号: RGPIN-2016-06281
• 负责人: Li, Baochun
• 金额: $ 4.74万
• 依托单位: University of Toronto
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2016
• 资助国家: 加拿大
• 起止时间: 2016-01-01 至 2017-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=616208
• 关键词: Managing; Resources; across; Geo; Distributed

Both big data analytics and mobile cloud applications are increasingly geographically distributed, relying upon services from multiple datacenters around the world. Yet, wide-area inter-datacenter links are costly to build and to upgrade, and inter-datacenter bandwidth is bound to become a scarce resource in the near future. Traditional data analytics need to first move all raw data to one datacenter before they can be processed, which adversely affect their performance and are bandwidth intensive. Real-time streaming analytics and interactive mobile cloud applications such as video conferencing and messaging, on the other hand, require low latencies. As such, there is a pressing need to manage and allocate inter-datacenter resources at a fine granularity to meet the diverse performance targets from both data analytics and mobile cloud applications. In this program, we propose to design and implement a new software-defined inter-datacenter infrastructure, in order to optimize the utilization of inter-datacenter bandwidth. Operating in the application layer, our infrastructure provides a substrate and a public application programming interface (API) to support existing data analytic frameworks and interactive mobile cloud applications. It schedules data analytic tasks, routes data streams, and manages bandwidth allocations to application flows at a fine granularity, with the objective of utilizing scarce inter-datacenter bandwidth with optimal efficiency. Following the principles of software-defined networking, our infrastructure fully decouples the control and data planes, and makes task scheduling, flow routing, and traffic engineering decisions based on live analytics of network conditions. We will propose new algorithms to make these decisions in the controller, catering for the specific needs of each category of applications. For batch and streaming analytics applications, our new algorithms are designed to optimize task scheduling and assignment in multiple datacenters, so that both the volume and latency of inter-datacenter traffic can be optimized without prior knowledge of the communication patterns of data analytics jobs. For interactive mobile cloud applications that are bandwidth-intensive and latency-sensitive, our algorithms are designed to optimize resource allocations according to the delay sensitivity of competing application flows. To be more expressive, our infrastructure uses executable scripts to implement control plane decisions in the data plane. We plan to implement our new infrastructure in a real-world inter-datacenter testbed involving multiple datacenters worldwide. This program will lead to fundamentally new ways of using a large number of datacenters around the world to support big data analytics and mobile applications, both of which will soon to be critical and indispensable to businesses and consumers alike in Canada.

大数据分析和移动云应用程序都越来越多地分布在地理上，这取决于世界各地的多个数据中心的服务。然而，广阔的地区间链接的构建和升级成本高昂，而跨境的带宽必定会在不久的将来成为稀缺的资源。传统的数据分析需要首先将所有原始数据移至一个数据中心，然后才能处理它们，从而对其性能产生不利影响，并且是带宽密集的。另一方面，实时流媒体分析和交互式移动云应用程序（例如视频会议和消息传递）需要低潜伏期。因此，有迫切需要以精细的粒度管理和分配数据间资源，以满足来自数据分析和移动云应用程序的各种性能目标。 在此程序中，我们建议设计和实施一个新的软件定义的跨境基础架构，以优化使用跨性别者间带宽的利用。在应用程序层中运行，我们的基础架构提供了基板和公共应用程序编程接口（API），以支持现有的数据分析框架和交互式移动云应用程序。它计划数据分析任务，路由数据流并管理带宽分配以以良好的粒度流动，目的是利用稀缺的互-DateCenter带宽，并具有最佳的效率。遵循软件定义的网络的原则，我们的基础架构完全解除了控制和数据平面，并根据网络条件的实时分析来制定任务调度，流程路由和交通工程决策。 我们将提出新的算法，以在控制器中做出这些决定，以满足每个类别应用程序的特定需求。对于批处理和流分析应用程序，我们的新算法旨在优化多个数据中心中的任务调度和分配，以便可以在不知道数据分析作业的通信模式的情况下优化 - 数据中流量的音量和延迟。对于具有带宽密集型和延迟敏感的交互式移动云应用程序，我们的算法旨在根据竞争应用程序流的延迟敏感性来优化资源分配。为了更具表现力，我们的基础架构使用可执行的脚本在数据平面中实现控制平面决策。我们计划在全球涉及多个数据中心的现实世界间测试床上实施新的基础架构。该计划将导致从根本上使用全球大量数据中心来支持大数据分析和移动应用程序的新方法，这两者都将很快对加拿大的企业和消费者来说都是至关重要的，并且是必不可少的。