Co-Designing Distributed Applications with Datacenter Networks

科学突破越来越依赖于处理大量信息来发现数据中的重要模式和关系。这需要的处理能力远远超过一台机器。相反，工作被分区并分发到通过高速网络连接的机器集群中。不幸的是，集群中机器之间的通信流量可能会使部分网络不堪重负，从而导致网络拥塞，从而限制了处理应用程序的可伸缩性和性能。当处理应用程序部署在共享环境（如公共云）中时，来自其他应用程序的流量会使问题进一步复杂化。因此，由于网络资源利用效率低下，可能会延迟甚至停止科学进步。******提出的研究计划旨在通过与网络一起设计应用程序来解决这个问题，从而更有效地利用网络资源。这种方法使应用程序能够使用最新的网络信息来做出与应用程序相关的决策。例如，为了响应网络拥塞，应用程序可能决定与不同的目的地通信。应用程序还可以控制网络，以确保其流量在整个网络中均匀分布。类似地，网络可以利用应用程序信息动态地更改其网络拓扑。该网络可以通过重新配置其拓扑结构来增加高流量端点之间的网络容量，从而减少网络拥塞。作为本研究计划的一部分，我们将进行测量研究，以了解不同类别应用程序中网络拥塞的原因。我们将开发一个应用程序框架，以简化应用程序和网络的开发，并促进采用这种设计方法。我们还将开发一个网络操作系统，以便在同一网络上运行多个应用程序时提供资源管理、冲突解决和策略执行。******这项工作的结果可以通过解决与网络相关的可扩展性限制而使科学界受益。这使得部署更大的处理集群能够处理比目前可能的更多的数据，这可以帮助加拿大科学家取得下一个科学突破。通过研究可以显著降低集群和数据中心的网络基础设施成本的技术，这项工作还可以使加拿大商业部门受益。云提供商可以使用我们的网络操作系统安全地为其租户提供对其网络部分的控制。最后，通过将我们的系统开源并与加拿大工业伙伴密切合作，我们可以迅速将我们的技术转让给加拿大公司，这可以使他们在全球市场上具有显著的竞争优势。

Scientific breakthroughs increasingly rely on processing vast volumes of information to discover important patterns and relationships in the data. This requires far more processing power than is available on a single machine. Instead, the work is partitioned and distributed to a cluster of machines connected by a high-speed network. Unfortunately, the communication traffic between machines in a cluster can overwhelm parts of the network, resulting in network congestion that limits the scalability and performance of the processing application. The problem is further compounded by traffic from other applications when the processing application is deployed in a shared environment, such as in the public cloud. As a result, scientific progress may be delayed or even halted because of inefficient network resource utilization.******The proposed research program aims to address this problem by designing applications together with the network to make more efficient use of network resources. This approach enables applications to use up-to-date network information to make application-related decisions. For example, an application may decide to communicate with a different destination in response to network congestion. The application can also control the network to ensure that its traffic is spread evenly across the network. Similarly, the network can leverage application information to dynamically change its network topology. The network can reduce network congestion by reconfiguring its topology to increase its network capacity between high-traffic end-points. As part of this research program, we will perform measurement studies to understand the causes of network congestion in different classes of applications. We will develop an application framework to simplify the development of applications and networks and promote adoption of this design approach. We will also develop a network operating system to provide resource management, conflict resolution, and policy enforcement when running multiple applications on the same network.******The results of this work can benefit the scientific community by addressing network-related scalability limitations. This enables the deployment of larger processing clusters that can process more data than is currently possible, which can help Canadian scientists make the next scientific breakthrough. This work can also benefit the Canadian commercial sector by investigating technologies that can significantly reduce the network infrastructure cost of clusters and datacenters. Cloud providers can use our network operating system to safely provide their tenants control over segments of their networks. Finally, by open-sourcing our systems and working closely with Canadian industrial partners, we can rapidly transfer our technology to Canadian companies, which can give them a significant competitive advantage in the global market.