NetPM: Co-designing Data Management and Networking Principles for Persistent Memory

点击翻译按钮获取中文摘要

System software overheads are becoming problematic in both data centre and Internet-facing systems to offer high quality, predictable services at low costs, because the networks are becoming faster and faster, and so are storage devices. These overheads stem from achieving various networking or storage properties, such as integrity, durability, consistency and flexibility of data organization, over unreliable, dumb networks or storage hardware. Achieving these properties overwhelms the CPUs and results in underutilization of high-speed networks and storage devices, including emerging persistent memory (PM). This observation implies that the operators need to invest more compute hardware resources to ensure the desired service-level objectives. Even worse, although achieving the storage properties is costly, fewer opportunities of hardware offloading are available in PM devices than in block devices, which act as I/O peripherals. This project aims to address this problem by repurposing features available in the network stack or protocols, which often can be offloaded to the networking hardware, for achieving storage properties, so that the storage stacks can minimise their operations. The key insight here is that many properties required by the storage stack are, by definition, also available in the network stack or end-to-end protocols like TCP, and these properties could be repurposed particularly when storage data is resident in not the block devices, but the emerging byte-addressable PM devices whose data structures resemble in-memory data structures.Despite the simple high-level concept, this architecture is difficult to achieve. Since network and storage stacks have been designed in isolation, existing data structures are not reusable between these stacks, meaning that I must design new ones that meet both networking and storage requirements. For example, since the current networking data structures are ephemeral, they are neither named nor crash consistent. Further, since the current standard APIs and programming models are designed for separate operations for networking and storage access, I must design new, useful programming abstractions that effectively utilize the network-aided storage stack and then build concrete networked storage applications on top of these abstractions. Therefore, to achieve and verify my approach, I must design the whole software stack across a variety of components that include low-level packet I/O framework, persistent networking data structures, high-level APIs and the applications.The programme of the project sets off by enabling techniques to repurpose the networking features for implementing the storage properties, including designing data structures, algorithms and APIs. It will then design and implement concrete software stacks for networked storage systems, including file system, key-value store and transactional database, each of which has different requirements and target workloads. In addition to experimental evaluation of those new systems based on realistic workloads, the project will finally validate the results by building more complex applications, such as distributed machine learning for graph structured data.I have a proven track record of leading the research work in the most relevant area, which is an intersection between networking and OS, and believe I have an ambitious goal backed by a reasonable work and resource plan and partner.