SHF: CNS Core: Small: Server architecture optimizations for microsecond-scale RPCs

现代数据中心托管的在线服务被分解为跨越数千台服务器的多个软件层。服务器通过内部数据中心网络使用远程过程调用（Remote Procedure call, rpc）相互协调和通信。微服务不断提高生产力的软件架构趋势正在将已部署服务的软件分解推向极致，导致更频繁的服务器间通信和更细粒度的rpc，通常运行时间只有几微秒。随着每个RPC运行时的缩减，网络效率直接影响整个在线服务的性能：与网络相关的开销本来可以忽略不计，但由于每个RPC触发的应用程序级逻辑的细粒度特性而被放大。解决这一挑战的一种很有希望的方法是将每个服务器的nic（服务器计算资源和网络之间的网关）的角色从简单的RPC交付提升到主动RPC操作。从历史上看，NIC通过将传入数据包写入内存来不可知地传递传入数据包；这些数据包随后由CPU核心拾取进行处理，从而导致过多的数据移动、核间同步开销或核间负载不平衡。ROAr是一种新的服务器架构，针对微秒级rpc的有效处理进行了优化，其特点是网卡可以动态监控系统范围的行为，并在服务器的内存层次结构中智能地引导传入的rpc，包括直接放置在CPU核心的私有缓存中。面向rpc的协议允许NIC将其操作的抽象级别从网络数据包提升到rpc。，从数据块到触发一些计算的消息。向NIC公开的关于RPC将触发的计算的信息越多，NIC就能做出更好的RPC转向决策。在ROAr体系结构中，NIC监视传入的rpc，并做出许多新颖的决策，以明智地将它们分布在现代服务器的内存层次结构中并跨CPU内核。总的来说，ROAr的技术可以大大提高处理微秒级rpc的效率和性能。一个直接的结果是提高了部署在现代数据中心上的大量大规模在线服务的质量，这些服务大量使用这种rpc。因此，ROAr有可能影响未来服务器架构的设计。ROAr涉及广泛的硬件软件协同设计，打破了网络和计算之间传统的严格界限。NIC的角色从不经意地将传入RPC放置到服务器的内存层次结构中提升到主动RPC加速。NIC在RPC处理生命周期中的自然位置使其成为一种出色的代理，可以为优化的数据移动和减少延迟安排缓存层次结构。ROAr主要有三种机制。首先，它通过将所有传入的rpc保持在缓存层次结构中来减轻有害的内存带宽干扰，提前拒绝由于过多的正在进行的排队而预计将错过截止日期的请求。其次，ROAr通过考虑实时系统负载信息，为传入的rpc做出动态的核间平衡决策，并将rpc一直引导到所选CPU核心的私有缓存。第三，当RPC排队等待执行时，NIC预取RPC相应的指令和关键数据，从而加快RPC最终被内核拾取进行处理时的启动时间。这种预取的本质是新颖的，因为决策不是基于预测，而是基于先见之明：NIC对RPC从网络到达的早期了解。建议的研究包括理论建模、仿真和原型设计。将对各种RPC服务时间分布进行排队分析，以开发nic驱动的核间负载分布策略。将开发一个周期精确的ROAr仿真模型，以评估缓存内网络缓冲管理、rpc到核心转向和预取机制。最后，通过使用基于可编程fpga的网卡，将评估网卡驱动的负载平衡策略在具有离散网卡的现有架构上的适用性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern datacenters host online services that are decomposed into multiple software tiers spanning thousands of servers. Servers coordinate and communicate with each other using Remote Procedure Calls (RPCs) over the internal datacenter network. The ongoing productivity-boosting software architecture trend of microservices is pushing software decomposition of deployed services to the extreme, resulting in more frequent inter-server communication and finer-grained RPCs, often with runtimes of only a few microseconds. With shrinking per-RPC runtime, networking efficiency directly impacts the performance of an online service as a whole: networking-related overheads that would otherwise be negligible are amplified by the fine-grained nature of the application-level logic triggered per RPC. A promising approach to address this challenge is to promote the role of each server’s NIC—the gateway between a server’s compute resources and the network—from simple RPC delivery to active RPC manipulation. Historically, the NIC agnostically delivers incoming packets, by writing them into memory; the packets are later picked up by a CPU core for processing, resulting in excess data movement, inter-core synchronization overheads, or inter-core load imbalance. ROAr is a new server architecture optimized for efficient handling of microsecond-scale RPCs, featuring a NIC that dynamically monitors system-wide behavior and intelligently steers incoming RPCs within the server’s memory hierarchy, including direct placement in CPU cores’ private caches. An RPC-oriented protocol allows the NIC to raise the level of abstraction it operates on from network packets to RPCs—i.e., from data chunks to messages that trigger some computation. The more information exposed to the NIC about the computation an RPC will trigger, the better the RPC steering decision the NIC can make. In the ROAr architecture, the NIC monitors incoming RPCs and makes a number of novel decisions to judiciously distribute them within a modern server’s memory hierarchy and across CPU cores. Overall, ROAr’s techniques can drastically improve the efficiency and performance of handling microsecond-scale RPCs. A direct consequence is improved quality for a plethora of large-scale online services deployed on modern datacenters, which make heavy use of such RPCs. Therefore, ROAr has the potential to influence the design of future server architectures.ROAr involves extensive hardware-software co-design, breaking the conventionally rigid boundaries between network and compute. The NIC’s role is promoted from oblivious placement of incoming RPCs into a server’s memory hierarchy to active RPC acceleration. The NIC’s natural position in an RPC’s processing lifetime establishes it as an excellent agent to stage the cache hierarchy for optimized data movement and reduced latency. ROAr features three main mechanisms. First, it alleviates detrimental memory-bandwidth interference by keeping all incoming RPCs within the cache hierarchy, early-rejecting requests that are predicted to miss their deadline because of excessive ongoing queuing. Second, ROAr makes dynamic inter-core balancing decisions for incoming RPCs, by taking into account real-time system load information, and steers RPCs all the way to the private cache of the selected CPU core. Third, while an RPC is queued, waiting to be executed, the NIC prefetches the RPC’s corresponding instructions and critical data, thus accelerating the RPC’s startup time when it is eventually picked up by the core for processing. The nature of such prefetching is novel, as decisions are not based on predictions, but on prescience: the NIC’s early knowledge of an RPC’s arrival from the network. The proposed research involves theoretical modeling, simulation, and prototyping. Queuing analysis on a variety of RPC service-time distributions will be conducted to develop NIC-driven inter-core load distribution policies. A cycle-accurate simulation model of ROAr will be developed to evaluate in-cache network buffer management, RPC-to-core steering, and prefetching mechanisms. Finally, the applicability of NIC-driven load-balancing policies on existing architectures featuring discrete NICs will be evaluated, with the use of a programmable FPGA-based NIC.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.