III: Medium: Mining petabytes of data using cloud computing and a massively parallel cyberinstrument

越来越需要有效的方法来开采非常大的，即，PB级规模的数据集在科学，工程和商业的许多领域。该项目旨在设计，分析和实现一些基本的矩阵挖掘和图形挖掘操作，可扩展到PB级规模的输入。这些努力保证了从大量矩阵和图形中分析、可视化和提取信息的惊人增长的持续。项目利用伦斯勒独特的计算平台，其形式为大规模并行机（Blue Gene/Q），可访问约1.2 PB的存储，以及一个名为RAM存储加速器（RSA）的数据暂存层，具有512个计算节点和总共8 TB的快速RAM。该平台可配置为允许RSA级别的计算节点用于以类似云的方式预处理来自辅助存储的数据。该项目旨在设计和分析矩阵和图挖掘任务的近似算法，这些任务遵循迭代的两步方法：给定PB级数据，首先，使用计算成本低廉的方法来获得紧凑的数据草图，使用RSA层作为“云”，以便将其大小从PB级减少到TB级。在Blue Gene/Q上使用计算要求很高的方法处理所得到的数据草图。使用近似解迭代该过程，以提高草图的质量和近似保证。研究团队希望发布矩阵和图挖掘算法的软件和库，这些算法实现了PB级矩阵和图的两阶段方法。由此产生的工具将被应用于分析从计算机模拟的生物分子系统的动力学PB的数据。研究人员计划通过实习计划让其他机构的学生和研究人员参与设计，分析和开发所提出的方法。该项目还为RPI的研究生和本科生提供了更多的数据分析和高性能计算研究培训机会。研究结果将通过项目网站提供给学术界。

There is a growing need for effective approaches to mining very large, i.e., petabyte scale data sets in many areas of science, engineering, and business.The project aims to design, analyze, and implement a number of fundamental matrix-mining and graph-mining operations that are scalable to petabyte-sized inputs. Such efforts guarantee the continuation of the phenomenal growth in analyzing, visualizing, and extracting information from massive matrices and graphs. Project leverages Rensselaer's unique computing platform in the form of a massively parallel machine (a Blue Gene/Q) with access to approximately 1.2 petabytes of storage, as well as a data-staging layer, named the RAM Storage Accelerator (RSA) with 512 computational nodes and a a total of 8TBs of fast RAM. The platform is configurable to allow the computational nodes at the RSA level to be used to pre-process data from the secondary storage in a cloud-like fashion. The project aims design and analyze approximation algorithms for matrix and graph mining tasks that follow an iterative, two-step approach: given petabytescale data, first, using computationally inexpensive approaches to obtain compact data sketches using the RSA layer as a "cloud" in order to reduce their size from the petabyte scale to the terabyte scale. The resulting data sketches are processed using computationally demanding approaches on the Blue Gene/Q. This process is iterated using the approximate solutions in order to improve the quality of the sketches and the approximation guarantees. The research team expects to release software and libraries for matrix and graph mining algorithms that implement our two-phase approaches for PB-scale matrices and graphs. The resulting tools will be applied to the analysis of petabytes of data from computer simulations of the dynamics of biomolecular systems. The investigators plan to involve students and researchers from other institutions in the design, analysis, and development of the proposed methods through an internship program. The project also offers increased opportunities for research-based training in Data Analytics and High Performance Computing to graduate and undergraduate students at RPI. The results of the research will be made available to the academic community through the project web site.