III: Medium: Mining petabytes of data using cloud computing and a massively parallel cyberinstrument

在科学、工程和商业的许多领域，越来越需要有效的方法来挖掘非常大的数据集，即PB级的数据集。该项目旨在设计、分析和实现许多基本的矩阵挖掘和图挖掘操作，这些操作可扩展到PB级的输入。这些努力保证了在分析、可视化和从海量矩阵和图形中提取信息方面的惊人增长的持续。该项目利用了Rensselaer独特的计算平台，其形式为可访问约1.2 PB存储的大规模并行机(Blue gene/Q)，以及名为RAM存储加速器(RSA)的数据登台层，该层具有512个计算节点和总计8TB的快速RAM。该平台可配置为允许使用RSA级别的计算节点以类似云的方式对来自辅助存储的数据进行预处理。该项目旨在为矩阵和图形挖掘任务设计和分析近似算法，这些任务遵循迭代的两步方法：给定PB级数据，首先使用计算成本低的方法，使用RSA层作为“云”来获得紧凑的数据草图，以便将其大小从PB级减少到TB级。所得到的数据草图使用蓝色基因/Q上计算要求高的方法来处理。该过程使用近似解来迭代，以提高草图的质量和近似保证。研究团队希望发布用于矩阵和图形挖掘算法的软件和库，这些算法实现了我们针对PB级矩阵和图形的两阶段方法。由此产生的工具将被应用于分析来自生物分子系统动力学的计算机模拟的PB数据。研究人员计划让来自其他机构的学生和研究人员通过实习计划参与拟议方法的设计、分析和开发。该项目还为RPI的研究生和本科生提供了更多的数据分析和高性能计算方面的研究性培训机会。研究结果将通过项目网站提供给学术界。

There is a growing need for effective approaches to mining very large, i.e., petabyte scale data sets in many areas of science, engineering, and business.The project aims to design, analyze, and implement a number of fundamental matrix-mining and graph-mining operations that are scalable to petabyte-sized inputs. Such efforts guarantee the continuation of the phenomenal growth in analyzing, visualizing, and extracting information from massive matrices and graphs. Project leverages Rensselaer's unique computing platform in the form of a massively parallel machine (a Blue Gene/Q) with access to approximately 1.2 petabytes of storage, as well as a data-staging layer, named the RAM Storage Accelerator (RSA) with 512 computational nodes and a a total of 8TBs of fast RAM. The platform is configurable to allow the computational nodes at the RSA level to be used to pre-process data from the secondary storage in a cloud-like fashion. The project aims design and analyze approximation algorithms for matrix and graph mining tasks that follow an iterative, two-step approach: given petabytescale data, first, using computationally inexpensive approaches to obtain compact data sketches using the RSA layer as a "cloud" in order to reduce their size from the petabyte scale to the terabyte scale. The resulting data sketches are processed using computationally demanding approaches on the Blue Gene/Q. This process is iterated using the approximate solutions in order to improve the quality of the sketches and the approximation guarantees. The research team expects to release software and libraries for matrix and graph mining algorithms that implement our two-phase approaches for PB-scale matrices and graphs. The resulting tools will be applied to the analysis of petabytes of data from computer simulations of the dynamics of biomolecular systems. The investigators plan to involve students and researchers from other institutions in the design, analysis, and development of the proposed methods through an internship program. The project also offers increased opportunities for research-based training in Data Analytics and High Performance Computing to graduate and undergraduate students at RPI. The results of the research will be made available to the academic community through the project web site.