Contiguitas: The Pursuit of Physical Memory Contiguity in Datacenters
Contiguitas: The Pursuit of Physical Memory Contiguity in Datacenters
复制标题
DOI：
10.1145/3579371.3589079
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Kaiyang Zhao;Kaiwen Xue;Ziqi Wang-;Dan Schatzberg;Leon Yang;Antonis Manousis;Johannes Weiner;Rik Van Riel;Bikash Sharma;Chunqiang Tang;Dimitrios Skarlatos
中科院分区：
文献类型：
--
作者：
Kaiyang Zhao;Kaiwen Xue;Ziqi Wang-;Dan Schatzberg;Leon Yang;Antonis Manousis;Johannes Weiner;Rik Van Riel;Bikash Sharma;Chunqiang Tang;Dimitrios Skarlatos
新兴数据中心应用对内存需求的持续增长，加剧了虚拟内存的可扩展性瓶颈。然而，在物理内存连续性困境得到解决之前，降低地址转换带来的过高开销仍将困难重重。为解决这一问题，本文提出了Contiguitas，这是一种对操作系统和硬件中内存管理的全新设计，可提供充足的物理内存连续性。我们发现，Meta数据中心内存碎片化的主要原因是不可移动的内存分配分散在地址空间中，阻碍了大的连续内存块的形成。为从设计上提供充足的物理内存连续性，Contiguitas首先将常规可移动内存分配与不可移动内存分配区分开来，将它们置于物理内存中两个不同的连续区域，并根据内存需求动态调整这两个区域的边界。大幅减少不可移动内存分配颇具挑战性，因为大多数不可移动页面仅靠软件无法移动，原因是为进行迁移而阻止对页面的访问是不可行的。此外，页面迁移成本高昂，因为这需要较长的停机时间来（a）执行与受害转换后备缓冲器（TLB）数量极不匹配的TLB清除操作，以及（b）复制页面。为此，Contiguitas通过在末级缓存中引入硬件扩展，消除了不可移动内存分配的主要来源，从而即使在页面仍在使用时，也能实现不可移动页面的透明且高效迁移。我们将Contiguitas的操作系统组件集成到Linux内核中，并在Meta数据中心的生产环境中进行实验。结果表明，Contiguitas的操作系统组件成功限制了不可移动内存分配，将分散在地址空间中的平均31% 的不可移动2MB块大幅减少至仅7% 集中在不可移动区域，从而显著提升了性能。具体而言，对于三项主要生产服务，Contiguitas在部分碎片化服务器上实现了2 - 9% 的端到端性能提升，在高度碎片化服务器上实现了7 - 18% 的提升，而高度碎片化服务器几乎占Meta服务器总量的四分之一。我们进一步通过全系统模拟展示了Contiguitas硬件扩展的有效性。评估显示，Contiguitas - HW能够高效迁移不可移动内存分配，与受害TLB数量的扩展性良好，且不影响应用程序性能。我们目前正在将Contiguitas纳入Linux上游。
The unabating growth of the memory needs of emerging datacenter applications has exacerbated the scalability bottleneck of virtual memory. However, reducing the excessive overhead of address translation will remain onerous until the physical memory contiguity predicament gets resolved. To address this problem, this paper presents Contiguitas, a novel redesign of memory management in the operating system and hardware that provides ample physical memory contiguity. We identify that the primary cause of memory fragmentation in Meta's datacenters is unmovable allocations scattered across the address space that impede large contiguity from being formed. To provide ample physical memory contiguity by design, Contiguitas first separates regular movable allocations from unmovable ones by placing them into two different continuous regions in physical memory and dynamically adjusts the boundary of the two regions based on memory demand. Drastically reducing unmovable allocations is challenging because the majority of unmovable pages cannot be moved with software alone given that access to the page cannot be blocked for a migration to take place. Furthermore, page migration is expensive as it requires a long downtime to (a) perform TLB shootdowns that scale poorly with the number of victim TLBs, and (b) copy the page. To this end, Contiguitas eliminates the primary source of unmovable allocations by introducing hardware extensions in the last-level cache to enable the transparent and efficient migration of unmovable pages even while the pages remain in use. We build the operating system component of Contiguitas into the Linux kernel and run our experiments in a production environment at Meta's datacenters. Our results show that Contiguitas's OS component successfully confines unmovable allocations, drastically reducing unmovable 2MB blocks from an average of 31% scattered across the address space down to 7% confined in the unmovable region, leading to significant performance gains. Specifically, we show that for three major production services, Contiguitas achieves end-to-end performance improvements of 2--9% for partially fragmented servers, and 7--18% for highly fragmented servers, which account for nearly a quarter of Meta's fleet. We further use full-system simulations to demonstrate the effectiveness of the hardware extensions of Contiguitas. Our evaluation shows that Contiguitas-HW enables the efficient migration of unmovable allocations, scales well with the number of victim TLBs, and does not affect application performance. We are currently in the process of upstreaming Contiguitas into Linux.