Co-Designing Distributed Applications with Datacenter Networks

科学突破越来越依赖于处理大量信息，以发现数据中的重要模式和关系。这需要的处理能力远远超过单台机器的处理能力。相反，工作被划分并分发到由高速网络连接的机器集群。不幸的是，集群中机器之间的通信流量可能会淹没网络的某些部分，导致网络拥塞，从而限制了处理应用程序的可扩展性和性能。当处理应用被部署在共享环境中（诸如在公共云中）时，来自其他应用的流量进一步加剧了该问题。因此，科学进步可能会因为网络资源利用效率低下而延迟甚至停止。拟议的研究计划旨在通过设计应用程序与网络一起更有效地利用网络资源来解决这个问题。这种方法使应用程序能够使用最新的网络信息来做出与应用程序相关的决策。例如，应用可以响应于网络拥塞而决定与不同的目的地通信。应用程序还可以控制网络，以确保其流量均匀地分布在整个网络中。类似地，网络可以利用应用信息来动态地改变其网络拓扑。网络可以通过重新配置其拓扑结构来增加其在高流量端点之间的网络容量，从而减少网络拥塞。作为这项研究计划的一部分，我们将进行测量研究，以了解不同类别的应用程序中网络拥塞的原因。我们将开发一个应用程序框架，以简化应用程序和网络的开发，并促进采用这种设计方法。我们还将开发一个网络操作系统，以便在同一网络上运行多个应用程序时提供资源管理、冲突解决和策略执行。这项工作的结果可以通过解决网络相关的可扩展性限制而使科学界受益。这使得能够部署更大的处理集群，可以处理比目前更多的数据，这可以帮助加拿大科学家取得下一个科学突破。这项工作还可以使加拿大的商业部门受益，因为它调查了可以大大降低集群和电信中心的网络基础设施成本的技术。云提供商可以使用我们的网络操作系统来安全地为其租户提供对其网络部分的控制。最后，通过开放我们的系统，并与加拿大工业伙伴密切合作，我们可以迅速将我们的技术转让给加拿大公司，这可以使他们在全球市场上具有显著的竞争优势。

Scientific breakthroughs increasingly rely on processing vast volumes of information to discover important patterns and relationships in the data. This requires far more processing power than is available on a single machine. Instead, the work is partitioned and distributed to a cluster of machines connected by a high-speed network. Unfortunately, the communication traffic between machines in a cluster can overwhelm parts of the network, resulting in network congestion that limits the scalability and performance of the processing application. The problem is further compounded by traffic from other applications when the processing application is deployed in a shared environment, such as in the public cloud. As a result, scientific progress may be delayed or even halted because of inefficient network resource utilization.******The proposed research program aims to address this problem by designing applications together with the network to make more efficient use of network resources. This approach enables applications to use up-to-date network information to make application-related decisions. For example, an application may decide to communicate with a different destination in response to network congestion. The application can also control the network to ensure that its traffic is spread evenly across the network. Similarly, the network can leverage application information to dynamically change its network topology. The network can reduce network congestion by reconfiguring its topology to increase its network capacity between high-traffic end-points. As part of this research program, we will perform measurement studies to understand the causes of network congestion in different classes of applications. We will develop an application framework to simplify the development of applications and networks and promote adoption of this design approach. We will also develop a network operating system to provide resource management, conflict resolution, and policy enforcement when running multiple applications on the same network.******The results of this work can benefit the scientific community by addressing network-related scalability limitations. This enables the deployment of larger processing clusters that can process more data than is currently possible, which can help Canadian scientists make the next scientific breakthrough. This work can also benefit the Canadian commercial sector by investigating technologies that can significantly reduce the network infrastructure cost of clusters and datacenters. Cloud providers can use our network operating system to safely provide their tenants control over segments of their networks. Finally, by open-sourcing our systems and working closely with Canadian industrial partners, we can rapidly transfer our technology to Canadian companies, which can give them a significant competitive advantage in the global market.