SPX: Collaborative Research: Mongo Graph Machine (MGM): A Flash-Based Appliance for Large Graph Analytics

我们生活在大数据时代。在许多问题领域，如数据挖掘、机器学习、科学计算和社会网络研究中，数据处理实体对之间的关系，并由称为图的数据结构表示。今天感兴趣的图形可能有数千亿个实体，以及这些实体之间的数万亿个关系。大规模的图形处理通常在数据中心完成，这些数据中心是由大量耗电的计算机组成的。Mongo Graph Machine （MGM）项目将探索一种不同的解决方案，即出核处理（out- core processing）。在这个系统中，图形将存储在比dram更便宜、密度更大、温度更低的闪存中，并使用称为fpga的专用电路与传统处理器的组合进行处理。将开发一种编程系统来对最终用户隐藏这种复杂性。最终的系统将小到可以放在桌子下面，并且在提供强大的图形处理能力的同时显着提高了能效。MGM项目将通过使用基于带有FPGA的NAND闪存芯片的存储加速来解决存储和处理极端规模图形的问题。使用当前的NAND技术，一台机器可以容纳1tb到16tb的闪存。这种配置提供了存储非常大的图形所需的闪存和使闪存带宽饱和所需的计算能力。为了解决该架构的编程问题，该项目将开发编译器技术和FPGA加速器，允许开发人员在高级编程模型中编写应用程序，将其留给系统利用并行性并优化闪存访问特性的内存访问。软件系统将基于伽罗瓦系统，该系统已被证明可以在大型共享内存机器上扩展到数百个处理器。

We live in the age of big data. In many problem domains such as data-mining, machine learning, scientific computing, and the study of social networks, the data deals with relationships between pairs of entities, and is represented by a data structure called a graph. Graphs of interest today may have hundreds of billions of entities, and trillions of relationships between these entities. Large-scale graph processing is typically done in data-centers which are huge clusters of power hungry computers. The proposed Mongo Graph Machine (MGM) project will explore a different solution known as out-of-core processing. In this system, graphs will be stored in flash memory, which is much cheaper, denser and cooler than DRAMs, and processed using a combination of specialized circuits called FPGAs in tandem with a conventional processor. A programming system will be developed to hide this complexity from the end-user. The resulting system will be small enough to fit under a desk and dramatically more energy-efficient while providing powerful graph processing capability.The MGM project will address the problem of storing and processing extreme-scale graphs by using in-storage acceleration based on NAND flash chips with an attached FPGA. A single machine can accommodate 1 TB to 16 TBs of flash memory using current NAND technology. This configuration provides the flash storage necessary to store very large graphs and the computational power necessary to saturate the bandwidth of the flash. To address the programming problem for this architecture, the project will develop compiler technology and FPGA accelerators that will permit developers to write applications in the high-level programming model, leaving it to the system to exploit parallelism and optimize memory accesses for the access characteristics of flash storage. The software system will be based on the Galois system, which has been shown to scale to hundreds of processors on large shared-memory machines.