Collaborative Research: CCF Core: Small: User-transparent Data Management for Persistence and Crash-consistency in Non-volatile Memories

非易失性存储器是一种计算机存储器，可以在断电或系统崩溃时保留存储的数据。与传统的易失性存储器相比，非易失性存储器具有大容量和低能耗的特点，长期以来一直被认为是构建高性能计算、机器学习和嵌入式系统等关键领域中大规模、低成本、高能效和可恢复应用的理想解决方案。尽管非易失性存储器可以作为商业存储器芯片使用，并提供了许多承诺，但它尚未在生产系统中广泛采用。主要障碍是难以确保数据及时和正确地写入非易失性存储器，使其在崩溃后恢复到一致的状态。目前，应用程序开发人员背负着将遗留应用程序移植到非易失性存储器的负担，这既繁琐又容易出错。该项目旨在为用户透明的持久性和崩溃一致性建立一个通用框架，允许未经修改的遗留应用程序使用非易失性存储器高效而正确地运行。该项目的成功将有助于释放非易失性存储器的全部潜力，并使其更容易采用。这项研究还将为未来混合、分散存储系统的数据管理提供有价值的见解。此外，该项目还包括指导博士生、吸引少数族裔学生、课程开发和K12外展活动。该项目将非易失性内存集成到内存管理中的页面/缓冲区缓存中--即连接字节可寻址内存和后备内存设备视图的抽象--为用户空间程序提供持久性和崩溃一致性，而无需或几乎没有用户参与。挑战在于1)如何拦截程序更新并将其重定向到非易失性存储器以进行持久化；2)如何正确地对更新进行排序并确保更新原子性以保证崩溃一致性；3)如何在不引起明显开销或性能下降的情况下将非易失性存储器有效地集成到页面/缓冲区缓存管理中。该项目通过持久化三种类型的程序数据-文件备份数据、动态分配的应用程序内存和用于虚拟内存管理的程序元数据(如页表)来解决这些挑战，并探索各种软件和硬件技术，如写入时复制、撤消日志记录、阴影分页和扩展页表，以实现高效的崩溃一致性。该项目促进了对易失性和非易失性存储器的混合存储器管理的理解，同时实现了高可用性、良好的向后兼容性和高效率。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Non-volatile memory is a type of computer memory that can retain stored data upon a power loss or system crash. Due to its large capacity and low energy footprint compared to traditional volatile memory, non-volatile memory has long been envisioned as an ideal solution for building large-scale, cost-effective, energy-efficient, and recoverable applications in many critical domains, including high-performance computing, machine learning, and embedded systems. Although non-volatile memory is available as commercial memory chips and offers numerous promises, it has not yet been widely adopted in production systems. The major obstacle is the difficulty to ensure that data is timely and correctly written to non-volatile memory, allowing it to be restored to a consistent state after a crash. Currently, application developers carry the burden of porting legacy applications to non-volatile memory, which is tedious and error-prone. This project seeks to establish a generic framework for user-transparent persistence and crash consistency that allows unmodified legacy applications to run efficiently and correctly with non-volatile memory. The success of this project will help unleash the full potential of non-volatile memory and make it easier to adopt. The research will also provide valuable insights into data management in future hybrid, disaggregated memory systems. In addition, this project involves mentoring Ph.D. students, engaging minority students, course development, and K12 outreach activities. This project integrates non-volatile memory into the page/buffer cache in memory management – i.e., an abstraction that bridges the view of byte-addressable memory and a backing memory device -- to provide persistence and crash consistency to user-space programs with no or little user involvement. The challenges lie in 1) how to intercept program updates and redirect them to non-volatile memory for persistence; 2) how to properly order the updates and ensure update atomicity to guarantee crash consistency; 3) how to efficiently integrate non-volatile memory into page/buffer cache management without incurring noticeable overhead or performance degradation. This project addresses these challenges by focusing on persisting three types of program data – file-backed data, dynamically allocated application memory, and program metadata for virtual memory management, such as page tables, and exploring various software and hardware techniques, such as copy-on-write, undo logging, shadow paging, and extended page tables, for each data type to achieve efficient crash consistency. This project advances the understanding of hybrid memory management for volatile and non-volatile memories while simultaneously achieving high usability, good backward compatibility, and high efficiency.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.