Storage and Memory Management for Data-Intensive Computing

数据密集型计算的存储和内存管理

• 批准号: RGPIN-2018-06391
• 负责人: DemkeBrown, Angela
• 金额: $ 2.99万
• 依托单位: University of Toronto
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2020
• 资助国家: 加拿大
• 起止时间: 2020-01-01 至 2021-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=712799
• 关键词: Storage; Memory; Management; Data; Intensive

Computation is increasingly moving into data centres with the widespread adoption of the Cloud computing model, while data centre computing itself is being radically transformed by the availability of very large main memories, the commodification of new hardware technologies such as Infiniband networking with Remote Direct Memory Access (RDMA) capabilities, and high capacity solid state disk (SSD) storage. In this environment, a rack of compute nodes has become the new data centre building block, known as rack-scale computing. Also, non-volatile memory is becoming a commercial reality with the introduction of Intel's 3D XPoint technology. This hardware revolution has profound implications for systems-level software, especially the storage and memory management layers, as old performance bottlenecks are eliminated and new ones emerge. Hardware advances and distributed data processing frameworks are fueling rapid advances in data analytics and machine learning, enabling applications in automated language translation, image understanding and autonomous vehicles (to name just a few). Today's frameworks are often scalable but highly inefficient, squandering the potential of new hardware with excessive software overheads. My research explores ways to manage the burgeoning wealth of hardware resources, enabling future data-intensive analytics applications to operate at larger scales, more efficiently, and with greater reliability. We summarize three specific avenues to achieve this goal. 1. A unified rack-scale persistent memory storage system: Existing research on persistent memory storage improves on traditional file systems designed for slow, block-based devices, but it does not consider issues of allocation, access control, and consistency when scaling beyond a single node. Future rack-scale computing will demand a scalable, parallel and distributed file system that exposes the performance of emerging persistent memory across a full rack of nodes. 2. Memory-locality aware scheduling for data analytics frameworks: Many large-scale data analytics applications use Apache Spark or similar computing frameworks, which are designed with the assumption that data is stored on slow-to-access disks. Disk locality is of decreasing importance when nodes in a rack are connected by high-speed networks. We are exploring ways to exploit information about what parts of the input data are already in memory at each node, making smarter task placement possible. 3. In-memory distributed spatiotemporal data analytics: The rise of sensors and GPS-equipped mobile devices is driving rapid growth of spatiotemporal data, enabling novel applications that require sophisticated spatiotemporal data analytics. The expanding data volume means these datasets may soon exceed the capacity of even large-memory servers, however, requiring new techniques for in-memory rack-scale processing.

随着云计算模型的广泛采用，计算正越来越多地转移到数据中心，而数据中心计算本身正在被非常大的主存储器的可用性、新硬件技术的商品化（如具有远程直接内存访问（RDMA）功能的Infiniband网络和高容量固态磁盘（SSD）存储）彻底改变。在这种环境中，计算节点的机架已成为新的数据中心构建块，称为机架级计算。此外，随着英特尔3D XPoint技术的引入，非易失性存储器正在成为商业现实。这种硬件革命对系统级软件，特别是存储和内存管理层具有深远的影响，因为旧的性能瓶颈被消除了，新的性能瓶颈出现了。