SHF: Small: Collaborative Research: Modeling and Analyzing Big Data on Peta- and Exascale Distributed Systems Supported by MapReduce Methodologies

目前的千万亿次平台可以执行大规模模拟，并以前所未有的速度生成大量数据。随着亿亿次平台的引入，这些比率预计将增加。越来越多的数据的产生给那些努力分析、整理和选择有科学意义的结果的科学家带来了新的挑战。当非常大量的数据记录位于分布式存储器系统中的大量节点上时，即使是少量的比较也可能是昂贵的甚至是不可能的。因此，需要新的方法来分析大规模的科学数据集。本项目的目标是开发一种变革性的分析方法，以分布式的方式在当今的千万亿次系统和未来的亿亿次系统上对大型科学数据集的属性进行建模。研究活动包括（1）通过使用空间缩减技术以并行方式对嵌入分布式数据中的属性进行编码的新算法的设计;（2）通过使用分布式范例（如MapReduce）对这些属性进行聚类和分类的新算法的设计;（3）将算法部署到结构生物学和天文学的不同数据集;以及（4）针对新兴存储技术的结果性能和准确性的算法的调整。该分析方法将为科学界提供基础设施和工具，以确定可用于预测类别成员资格的特征;在数据集中找到经常性模式;并从特定特征或属性中确定类别成员资格。通过以可扩展的方式有效和准确地捕获科学信息，这些基础设施和仪器将打破数据集中的传统限制，使科学家能够克服与所考虑的数据的完全分布式性质相关的困难。该项目的教育部分促进了计算建模和分析技术以及数据方面的培训和学习。通过让本科生和研究生参与研究活动，并将大数据分析整合到特拉华州大学的本科课程中，我们将为学生提供密集的算法和平台。该项目中开发的基于研究的教育材料将通过项目门户网站以及XSEDE和超级计算会议上的教程提供给科学界。

Current petascale platforms can perform large-scale simulations and generate massive amounts of data at unprecedented rates. These rates are expected to increase as exascale platforms are introduced. The generation of more and more data presents new challenges for scientists who struggle with the analysis, sorting, and selection of scientifically meaningful results. When very large amounts of data records are located across a large number of nodes in a distributed memory system, even a small number of comparisons can be costly or even impossible. Therefore, new methodologies are necessary to analyze large scientific datasets at scale.The goal of this project is to develop a transformative analysis method to model the properties of large scientific datasets in a distributed manner on petascale systems today and exascale systems in the future. The research activity includes (1) the design of new algorithms for encoding properties embedded in distributed data in a parallel manner by using space reduction techniques; (2) the design of new algorithms for clustering and classifying these properties by using distributed paradigms such as MapReduce; (3) the deployment of the algorithms for diverse datasets in structural biology and astronomy; and (4) the tuning of the algorithms for both result performance and accuracy on emerging storage technologies. The analysis method will provide the scientific community with infrastructures and instrumentations to identify features that can be used to predict class memberships; find recurrent patterns in datasets; and identify class memberships from a specific feature or property. By effectively and accurately capturing scientific information in a scalable manner, these infrastructures and instrumentations will break the traditional constraint of data centralization and allow scientists to overcome the difficulties associated with the fully distributed nature of the data considered.The project's educational component promotes training and learning in computational modeling and analysis techniques as well as data-intensive algorithms and platforms by involving undergraduate and graduate students in research activities and integrating big data analytics into the undergraduate curriculum at the University of Delaware. The research-based educational materials developed in this project will be made available to the scientific community through the project portal and through tutorials at XSEDE and Supercomputing (SC) conferences.