A new server I/O architecture for high speed networks
A new server I/O architecture for high speed networks
复制标题
DOI：
10.1109/hpca.2011.5749734
复制
发表时间：
2011-02
期刊：
影响因子：
--
通讯作者：
Guangdeng Liao;Xia Zhu;L. Bhuyan
中科院分区：
文献类型：
--
作者：
Guangdeng Liao;Xia Zhu;L. Bhuyan
传统的架构设计通常集中在CPU上，并且通常与I/O考虑因素分离。它们对于带宽为10 Gbps及以上的高速网络处理效率低下。主流服务器上的长延迟I/O互连也使NIC设计大大复杂化。在本文中，我们从细粒度驱动程序和操作系统插装开始，以充分了解主流服务器上超过10 GbE的网络处理开销。我们得到了几个新的发现：1）除了以前的工作所确定的数据复制，驱动程序和缓冲区释放是两个意想不到的主要开销（高达54%）; 2）开销的主要来源是存储器停顿，与套接字缓冲区（SKB）和页数据结构相关的数据是造成停顿的主要原因; 3）直接缓存访问（DCA）等流行的平台优化不足以解决网络处理瓶颈。出于研究的动机，我们提出了一种新的服务器I/O架构，其中DMA描述符管理转移到一个片上网络引擎（NEngine），和描述符扩展与信息有关的数据引起内存失速。NEngine依赖于数据查找并预加载数据来消除网络处理期间的停顿。此外，NEngine在缓存内实现了高效的数据包移动，以解决数据复制中的剩余问题。新的架构允许DMA引擎非常快速地访问描述符，并将数据包保存在CPU缓存中，而不是NIC缓冲区中，从而大大简化了存储。实验结果表明，新的服务器I/O架构将网络处理效率提高了47%，Web服务器吞吐量提高了14%，同时大幅降低了NIC硬件复杂度。
Traditional architectural designs are normally focused on CPUs and have been often decoupled from I/O considerations. They are inefficient for high-speed network processing with a bandwidth of 10Gbps and beyond. Long latency I/O interconnects on mainstream servers also substantially complicate the NIC designs. In this paper, we start with fine-grained driver and OS instrumentation to fully understand the network processing overhead over 10GbE on mainstream servers. We obtain several new findings: 1) besides data copy identified by previous works, the driver and buffer release are two unexpected major overheads (up to 54%); 2) the major source of the overheads is memory stalls and data relating to socket buffer (SKB) and page data structures are mainly responsible for the stalls; 3) prevailing platform optimizations like Direct Cache Access (DCA) are insufficient for addressing the network processing bottlenecks. Motivated by the studies, we propose a new server I/O architecture where DMA descriptor management is shifted from NICs to an on-chip network engine (NEngine), and descriptors are extended with information about data incurring memory stalls. NEngine relies on data lookups and preloads data to eliminate the stalls during network processing. Moreover, NEngine implements efficient packet movement inside caches to address the remaining issues in data copy. The new architecture allows DMA engine to have very fast access to descriptors and keeps packets in CPU caches instead of NIC buffers, significantly simplifying NICs. Experimental results demonstrate that the new server I/O architecture improves the network processing efficiency by 47% and web server throughput by 14%, while substantially reducing the NIC hardware complexity.