SHF: Small: Collaborative Research: Modeling and Analyzing Big Data on Peta- and Exascale Distributed Systems supported by MapReduce Methodologies

目前的千兆级平台可以执行大规模模拟，并以前所未有的速度生成大量数据。随着百亿亿级平台的引入，这些速率预计会增加。越来越多的数据的产生给科学家们提出了新的挑战，他们正在努力分析、整理和选择有科学意义的结果。当非常大量的数据记录位于分布式内存系统中的大量节点时，即使是少量的比较也可能是昂贵的，甚至是不可能的。因此，需要新的方法来分析大规模的科学数据集。该项目的目标是开发一种变革性的分析方法，以分布式方式在当前的千万亿级系统和未来的百亿亿级系统上对大型科学数据集的属性进行建模。研究活动包括：(1)利用空间缩减技术，以并行方式设计嵌入分布式数据的编码属性的新算法；(2)利用MapReduce等分布式范式设计新的属性聚类和分类算法；(3)结构生物学和天文学中不同数据集的算法部署；(4)在新兴存储技术上对算法进行优化，以提高结果的性能和准确性。该分析方法将为科学界提供基础设施和仪器，以识别可用于预测类成员的特征；在数据集中发现循环模式；并从特定的特征或属性中识别类成员。通过以可扩展的方式有效和准确地捕获科学信息，这些基础设施和仪器将打破数据集中的传统限制，使科学家能够克服与所考虑的数据的完全分布式特性相关的困难。该项目的教育部分通过让本科生和研究生参与研究活动，并将大数据分析整合到特拉华大学的本科课程中，促进了计算建模和分析技术以及数据密集型算法和平台的培训和学习。在这个项目中开发的以研究为基础的教育材料将通过项目门户网站和XSEDE和超级计算会议上的教程提供给科学界。

Current petascale platforms can perform large-scale simulations and generate massive amounts of data at unprecedented rates. These rates are expected to increase as exascale platforms are introduced. The generation of more and more data presents new challenges for scientists who struggle with the analysis, sorting, and selection of scientifically meaningful results. When very large amounts of data records are located across a large number of nodes in a distributed memory system, even a small number of comparisons can be costly or even impossible. Therefore, new methodologies are necessary to analyze large scientific datasets at scale.The goal of this project is to develop a transformative analysis method to model the properties of large scientific datasets in a distributed manner on petascale systems today and exascale systems in the future. The research activity includes (1) the design of new algorithms for encoding properties embedded in distributed data in a parallel manner by using space reduction techniques; (2) the design of new algorithms for clustering and classifying these properties by using distributed paradigms such as MapReduce; (3) the deployment of the algorithms for diverse datasets in structural biology and astronomy; and (4) the tuning of the algorithms for both result performance and accuracy on emerging storage technologies. The analysis method will provide the scientific community with infrastructures and instrumentations to identify features that can be used to predict class memberships; find recurrent patterns in datasets; and identify class memberships from a specific feature or property. By effectively and accurately capturing scientific information in a scalable manner, these infrastructures and instrumentations will break the traditional constraint of data centralization and allow scientists to overcome the difficulties associated with the fully distributed nature of the data considered.The project's educational component promotes training and learning in computational modeling and analysis techniques as well as data-intensive algorithms and platforms by involving undergraduate and graduate students in research activities and integrating big data analytics into the undergraduate curriculum at the University of Delaware. The research-based educational materials developed in this project will be made available to the scientific community through the project portal and through tutorials at XSEDE and Supercomputing (SC) conferences.