BIGDATA: F: DKM: Collaborative Research: PXFS: ParalleX Based Transformative I/O System for Big Data

近几十年来，计算科学得到了发展，其中建模和数据分析对于科学和工程领域的新创新的探索、发现和改进至关重要。最近，这些技术被应用于艺术、社会、政治和其他传统上不太依赖高性能计算的领域。这种创新源于大约20年前对高性能并行和分布式架构的I/O（输入/输出）支持落后于纯计算速度的认识，进一步提高I/O的速度是一个既关键又相当困难的问题。在大型HPC机器上，当代I/O的核心障碍与延迟问题有关，这在很大程度上是由历史I/O模型的缺陷引起的，这与计算机完全是大型的、集中的、由许多分时程序共享的单处理器系统有关。为了在未来的硬件架构上提高I/O的可伸缩性，需要新的方法。该项目正在研究ParalleX的扩展，这是一种新的高度创新的并行执行模型。该扩展提供了一个强大的I/O接口，使研究人员能够为大数据应用程序创建高效的数据管理，发现和分析代码。这个新的扩展名为PXFS，它基于HPX（一种基于c++的parallelx实现）和OrangeFS（一种高性能并行文件系统）。驱动PXFS的研究目标是将HPX对象扩展到I/O空间中，以便对象变得持久，存储成为另一类内存，所有这些都作为单个虚拟地址空间访问，并由事件驱动的动态自适应计算环境进行管理。该方法的关键方面包括基于期货的同步、动态局部性管理、动态资源管理、分层名称空间和活动全局地址空间（AGAS）。PXFS的总体目标是通过统一名称空间及其管理来消除传统文件系统强加的编程划分，并最小化全局同步以支持异步并发性。研究方法是使用PXFS实现Map/Reduce应用程序框架，并评估其在性能和易用性方面的有效性。本项目在三所主要研究型大学进行，涉及本科生和研究生、博士后、高中教师及其学生。该项目包括来自功能基因组学领域的PI，作为领域科学专家，以便将开发工作集中在现实世界的问题上。参与该项目的研究生和博士后在这些领域进行培训，以促进了解大数据问题的两个方面的科学家。该项目吸引少数族裔参与，目的是激励他们从事计算机科学或基因组学方面的职业。该项目开发的软件是开放源代码的，并使用集成的源代码修订存储库、wiki和bug跟踪软件系统存档，此外还有附带文档的代码发布。

Recent decades have seen the development of computational science where modeling and data analysis are critical to exploration, discovery, and refinement of new innovations in science and engineering. More recently the techniques have been applied to arts, social, political and other fields less traditionally reliant on high performance computing. This innovation has grown out of realization some 20 years ago that I/O (input/output) support for high performance parallel and distributed architectures had lagged behind that of pure computational speed, and further that bring I/O up to speed was both critical, and a rather difficult problem. The core hurdle of contemporary I/O on large HPC machines relates to issues of latency in large parts caused by the deficiencies of the historical I/O model that was relevant when computers were exclusively large, centralized, single processor systems shared by many time-sharing programs. In order to improve I/O on scalability on future hardware architectures novel approaches are required.This project is conducting research on an extension of ParalleX, a new highly innovative parallel execution model. The extension provides a powerful I/O interface that allows researchers to create highly efficient data management, discovery, and analysis codes for Big Data applications. This new extension, known as PXFS, is based on HPX, an implementation of ParalleX based on C++, and OrangeFS, a high performance parallel file system. The research goal driving PXFS is to extend HPX objects into I/O space so that the objects become persistent and storage becomes another class of memory, all accessed as a single virtual address space and managed by an event driven dynamic adaptive computation environment. Critical aspects of this approach include futures-based synchronization, dynamic locality management, dynamic resource management, hierarchical name space, and an active global address space (AGAS). The overall goals of PXFS are to eliminate the division of programming imposed by conventional file system through the unification of name spaces and their management, and to minimize global synchronization in order to support asynchronous concurrency. The research methodology is to implement a Map/Reduce application framework using PXFS and evaluate its effectiveness in both performance and ease of use.This project is conducted at three major research universities involving undergraduate and graduate students, post-docs, and high-school teachers and their students. The project includes a PI from the functional genomics field acting as domain science expert in order to focus the development efforts on real world problems. Graduate students and post-docs involved in the project are trained in these areas to promote scientists who understanding both aspects of Big Data problems. The project engages under represented minorities with the goal to inspire them to pursue a career in computer science or genomics. The software developed by the project is available open-source and archived using an integrated source code revision repository, wiki, and bug tracking software system in addition to code releases with accompanying documentation.