III: Medium: Mining petabytes of data using cloud computing and a massively parallel cyberinstrument

在科学、工程和商业的许多领域，越来越需要有效的方法来挖掘非常大的数据集，即pb级的数据集。该项目旨在设计、分析和实现一些基本的矩阵挖掘和图形挖掘操作，这些操作可扩展到pb大小的输入。这样的努力保证了在分析、可视化和从大量矩阵和图形中提取信息方面的持续显著增长。该项目利用Rensselaer独特的计算平台，采用大规模并行机（Blue Gene/Q）的形式，可以访问大约1.2 pb的存储，以及一个名为RAM存储加速器（RSA）的数据分级层，具有512个计算节点和总共8tb的快速RAM。该平台是可配置的，允许使用RSA级别的计算节点以类似云的方式预处理来自二级存储的数据。该项目旨在设计和分析矩阵和图形挖掘任务的近似算法，这些算法遵循迭代的两步方法：给定pb级数据，首先，使用计算成本低廉的方法获得紧凑的数据草图，使用RSA层作为“云”，以便将其大小从pb级减少到tb级。所得到的数据草图使用Blue Gene/Q上的计算要求很高的方法进行处理。为了提高草图的质量和近似保证，该过程使用近似解进行迭代。研究团队希望发布矩阵和图形挖掘算法的软件和库，以实现我们对pb级矩阵和图形的两阶段方法。由此产生的工具将被应用于分析来自生物分子系统动力学的计算机模拟的pb级数据。研究人员计划通过一个实习项目，让其他机构的学生和研究人员参与设计、分析和开发所提出的方法。该项目还为RPI的研究生和本科生提供了更多的数据分析和高性能计算方面的研究培训机会。研究结果将通过该项目的网站提供给学术界。

There is a growing need for effective approaches to mining very large, i.e., petabyte scale data sets in many areas of science, engineering, and business.The project aims to design, analyze, and implement a number of fundamental matrix-mining and graph-mining operations that are scalable to petabyte-sized inputs. Such efforts guarantee the continuation of the phenomenal growth in analyzing, visualizing, and extracting information from massive matrices and graphs. Project leverages Rensselaer's unique computing platform in the form of a massively parallel machine (a Blue Gene/Q) with access to approximately 1.2 petabytes of storage, as well as a data-staging layer, named the RAM Storage Accelerator (RSA) with 512 computational nodes and a a total of 8TBs of fast RAM. The platform is configurable to allow the computational nodes at the RSA level to be used to pre-process data from the secondary storage in a cloud-like fashion. The project aims design and analyze approximation algorithms for matrix and graph mining tasks that follow an iterative, two-step approach: given petabytescale data, first, using computationally inexpensive approaches to obtain compact data sketches using the RSA layer as a "cloud" in order to reduce their size from the petabyte scale to the terabyte scale. The resulting data sketches are processed using computationally demanding approaches on the Blue Gene/Q. This process is iterated using the approximate solutions in order to improve the quality of the sketches and the approximation guarantees. The research team expects to release software and libraries for matrix and graph mining algorithms that implement our two-phase approaches for PB-scale matrices and graphs. The resulting tools will be applied to the analysis of petabytes of data from computer simulations of the dynamics of biomolecular systems. The investigators plan to involve students and researchers from other institutions in the design, analysis, and development of the proposed methods through an internship program. The project also offers increased opportunities for research-based training in Data Analytics and High Performance Computing to graduate and undergraduate students at RPI. The results of the research will be made available to the academic community through the project web site.