Collaborative Research: CCF Core: Small: User-transparent Data Management for Persistence and Crash-consistency in Non-volatile Memories

非易失性存储器是一种计算机存储器，它可以在断电或系统崩溃时保留存储的数据。与传统的易失性存储器相比，由于其大容量和低能耗，非易失性存储器一直被认为是在许多关键领域（包括高性能计算、机器学习和嵌入式系统）构建大规模、经济高效、节能和可恢复应用的理想解决方案。尽管非易失性存储器可以作为商业存储器芯片使用，并提供了许多承诺，但它尚未在生产系统中广泛采用。主要障碍是难以确保数据及时正确地写入非易失性存储器，从而在崩溃后恢复到一致状态。目前，应用程序开发人员承担着将遗留应用程序移植到非易失性内存的重担，这既繁琐又容易出错。该项目旨在建立一个通用框架，用于用户透明的持久性和崩溃一致性，允许未经修改的遗留应用程序在非易失性内存中高效、正确地运行。该项目的成功将有助于释放非易失性存储器的全部潜力，并使其更易于采用。该研究还将为未来混合、分解存储系统的数据管理提供有价值的见解。此外，该项目还包括指导博士生、吸引少数民族学生、课程开发和K12外展活动。该项目将非易失性内存集成到内存管理中的页面/缓冲区缓存中——即，一种抽象，连接了字节可寻址内存和后备内存设备的视图——为用户空间程序提供持久性和崩溃一致性，而无需或很少需要用户参与。挑战在于1)如何拦截程序更新并将其重定向到非易失性存储器以实现持久性；2)如何正确排序更新，保证更新原子性，保证崩溃一致性；3)如何有效地将非易失性内存集成到页面/缓冲区缓存管理中，而不会引起明显的开销或性能下降。本项目解决这些挑战的方法是，专注于持久化三种类型的程序数据——文件支持的数据、动态分配的应用程序内存和用于虚拟内存管理（如页表）的程序元数据，并为每种数据类型探索各种软件和硬件技术，如写时复制、撤消日志记录、影子分页和扩展页表，以实现有效的崩溃一致性。该项目推进了对易失性和非易失性存储器的混合存储器管理的理解，同时实现了高可用性，良好的向后兼容性和高效率。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Non-volatile memory is a type of computer memory that can retain stored data upon a power loss or system crash. Due to its large capacity and low energy footprint compared to traditional volatile memory, non-volatile memory has long been envisioned as an ideal solution for building large-scale, cost-effective, energy-efficient, and recoverable applications in many critical domains, including high-performance computing, machine learning, and embedded systems. Although non-volatile memory is available as commercial memory chips and offers numerous promises, it has not yet been widely adopted in production systems. The major obstacle is the difficulty to ensure that data is timely and correctly written to non-volatile memory, allowing it to be restored to a consistent state after a crash. Currently, application developers carry the burden of porting legacy applications to non-volatile memory, which is tedious and error-prone. This project seeks to establish a generic framework for user-transparent persistence and crash consistency that allows unmodified legacy applications to run efficiently and correctly with non-volatile memory. The success of this project will help unleash the full potential of non-volatile memory and make it easier to adopt. The research will also provide valuable insights into data management in future hybrid, disaggregated memory systems. In addition, this project involves mentoring Ph.D. students, engaging minority students, course development, and K12 outreach activities. This project integrates non-volatile memory into the page/buffer cache in memory management – i.e., an abstraction that bridges the view of byte-addressable memory and a backing memory device -- to provide persistence and crash consistency to user-space programs with no or little user involvement. The challenges lie in 1) how to intercept program updates and redirect them to non-volatile memory for persistence; 2) how to properly order the updates and ensure update atomicity to guarantee crash consistency; 3) how to efficiently integrate non-volatile memory into page/buffer cache management without incurring noticeable overhead or performance degradation. This project addresses these challenges by focusing on persisting three types of program data – file-backed data, dynamically allocated application memory, and program metadata for virtual memory management, such as page tables, and exploring various software and hardware techniques, such as copy-on-write, undo logging, shadow paging, and extended page tables, for each data type to achieve efficient crash consistency. This project advances the understanding of hybrid memory management for volatile and non-volatile memories while simultaneously achieving high usability, good backward compatibility, and high efficiency.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.