SPX: Collaborative Research: Mongo Graph Machine (MGM): A Flash-Based Appliance for Large Graph Analytics

我们生活在大数据时代。在许多问题领域，如数据挖掘、机器学习、科学计算和社会网络研究中，数据处理实体对之间的关系，并由称为图的数据结构来表示。今天感兴趣的图表可能有数千亿个实体，以及这些实体之间数万亿的关系。大规模的图形处理通常在数据中心完成，这些数据中心是耗电的计算机的巨大集群。拟议中的Mongo Graph Machine(MGM)项目将探索一种不同的解决方案，即内核外处理。在这个系统中，图形将被存储在闪存中，闪存比DRAM更便宜、更密集、更酷，并使用称为现场可编程门阵列的专门电路的组合与传统处理器一起处理。将开发一个编程系统，对最终用户隐藏这一复杂性。由此产生的系统将足够小，可以放在桌子下，在提供强大的图形处理能力的同时，大大提高了能效。米高梅项目将通过使用基于NAND闪存芯片和附加FPGA的存储加速来解决存储和处理极端规模图形的问题。使用当前的NAND技术，一台机器可以容纳1 TB到16 TB的闪存。这种配置提供了存储非常大的图形所需的闪存存储，以及使闪存带宽饱和所需的计算能力。为了解决该体系结构的编程问题，该项目将开发编译器技术和FPGA加速器，允许开发人员以高级编程模型编写应用程序，将利用并行性和针对闪存的访问特性优化存储器访问的任务留给系统。该软件系统将基于Galois系统，该系统已被证明可在大型共享内存机器上扩展到数百个处理器。

We live in the age of big data. In many problem domains such as data-mining, machine learning, scientific computing, and the study of social networks, the data deals with relationships between pairs of entities, and is represented by a data structure called a graph. Graphs of interest today may have hundreds of billions of entities, and trillions of relationships between these entities. Large-scale graph processing is typically done in data-centers which are huge clusters of power hungry computers. The proposed Mongo Graph Machine (MGM) project will explore a different solution known as out-of-core processing. In this system, graphs will be stored in flash memory, which is much cheaper, denser and cooler than DRAMs, and processed using a combination of specialized circuits called FPGAs in tandem with a conventional processor. A programming system will be developed to hide this complexity from the end-user. The resulting system will be small enough to fit under a desk and dramatically more energy-efficient while providing powerful graph processing capability.The MGM project will address the problem of storing and processing extreme-scale graphs by using in-storage acceleration based on NAND flash chips with an attached FPGA. A single machine can accommodate 1 TB to 16 TBs of flash memory using current NAND technology. This configuration provides the flash storage necessary to store very large graphs and the computational power necessary to saturate the bandwidth of the flash. To address the programming problem for this architecture, the project will develop compiler technology and FPGA accelerators that will permit developers to write applications in the high-level programming model, leaving it to the system to exploit parallelism and optimize memory accesses for the access characteristics of flash storage. The software system will be based on the Galois system, which has been shown to scale to hundreds of processors on large shared-memory machines.