Co-Designing Distributed Applications with Datacenter Networks

科学突破越来越依赖于处理海量信息来发现数据中的重要模式和关系。这需要的处理能力远远超过单台机器所能提供的能力。取而代之的是，工作被分区并分配到通过高速网络连接的机器集群。遗憾的是，集群中计算机之间的通信流量可能会淹没部分网络，导致网络拥塞，从而限制处理应用程序的可伸缩性和性能。当处理应用部署在共享环境中(例如在公共云中)时，来自其他应用的流量进一步加剧了问题。因此，科学进步可能会因为网络资源利用效率低下而延迟甚至停止。 拟议的研究计划旨在通过与网络一起设计应用程序来解决这一问题，以更有效地利用网络资源。此方法使应用程序能够使用最新的网络信息来做出与应用程序相关的决策。例如，应用程序可能决定与不同的目的地通信以响应网络拥塞。该应用程序还可以控制网络，以确保其流量在网络中均匀分布。同样，网络可以利用应用程序信息来动态更改其网络拓扑。网络可以通过重新配置其拓扑结构来增加其在高流量端点之间的网络容量，从而减少网络拥塞。作为此研究计划的一部分，我们将进行测量研究，以了解不同类别应用程序中网络拥塞的原因。我们将开发一个应用程序框架，以简化应用程序和网络的开发，并推动采用这种设计方法。我们还将开发一个网络操作系统，以便在同一网络上运行多个应用程序时提供资源管理、冲突解决和策略执行。 这项工作的结果可以通过解决与网络相关的可扩展性限制而使科学界受益。这使得能够部署更大的处理集群，可以处理比目前可能处理的更多的数据，这可以帮助加拿大科学家取得下一个科学突破。这项工作还可以通过研究可以显著降低集群和数据中心的网络基础设施成本的技术，使加拿大商业部门受益。云提供商可以使用我们的网络操作系统安全地为其租户提供对其网络段的控制。最后，通过开源我们的系统并与加拿大工业合作伙伴密切合作，我们可以迅速将我们的技术转让给加拿大公司，这可以使它们在全球市场上获得显著的竞争优势。

Scientific breakthroughs increasingly rely on processing vast volumes of information to discover important patterns and relationships in the data. This requires far more processing power than is available on a single machine. Instead, the work is partitioned and distributed to a cluster of machines connected by a high-speed network. Unfortunately, the communication traffic between machines in a cluster can overwhelm parts of the network, resulting in network congestion that limits the scalability and performance of the processing application. The problem is further compounded by traffic from other applications when the processing application is deployed in a shared environment, such as in the public cloud. As a result, scientific progress may be delayed or even halted because of inefficient network resource utilization. The proposed research program aims to address this problem by designing applications together with the network to make more efficient use of network resources. This approach enables applications to use up-to-date network information to make application-related decisions. For example, an application may decide to communicate with a different destination in response to network congestion. The application can also control the network to ensure that its traffic is spread evenly across the network. Similarly, the network can leverage application information to dynamically change its network topology. The network can reduce network congestion by reconfiguring its topology to increase its network capacity between high-traffic end-points. As part of this research program, we will perform measurement studies to understand the causes of network congestion in different classes of applications. We will develop an application framework to simplify the development of applications and networks and promote adoption of this design approach. We will also develop a network operating system to provide resource management, conflict resolution, and policy enforcement when running multiple applications on the same network. The results of this work can benefit the scientific community by addressing network-related scalability limitations. This enables the deployment of larger processing clusters that can process more data than is currently possible, which can help Canadian scientists make the next scientific breakthrough. This work can also benefit the Canadian commercial sector by investigating technologies that can significantly reduce the network infrastructure cost of clusters and datacenters. Cloud providers can use our network operating system to safely provide their tenants control over segments of their networks. Finally, by open-sourcing our systems and working closely with Canadian industrial partners, we can rapidly transfer our technology to Canadian companies, which can give them a significant competitive advantage in the global market.