BIGDATA: F: DKM: Collaborative Research: PXFS: ParalleX Based Transformative I/O System for Big Data

近几十年来，计算科学得到了发展，建模和数据分析对于探索、发现和改进科学和工程领域的新创新至关重要。最近，这些技术已经应用于艺术、社会、政治和其他传统上不太依赖高性能计算的领域。这项创新是在大约20年前认识到对高性能并行和分布式架构的I/O（输入/输出）支持落后于纯计算速度，并且进一步使I/O达到速度是至关重要的，也是一个相当困难的问题。在大型HPC机器上的当代I/O的核心障碍涉及由历史I/O模型的缺陷引起的大部分延迟问题，当计算机是由许多分时程序共享的专门的大型集中式单处理器系统时，这是相关的。为了提高I/O在未来硬件架构上的可扩展性，需要新的方法。本项目正在研究ParalleX的扩展，这是一种新的高度创新的并行执行模型。该扩展提供了强大的I/O接口，使研究人员能够为大数据应用程序创建高效的数据管理，发现和分析代码。 这个新的扩展名为PXFS，它基于HPX（一种基于C++的ParalleX实现）和OrangeFS（一种高性能并行文件系统）。 驱动PXFS的研究目标是将HPX对象扩展到I/O空间，使对象变得持久，存储成为另一类内存，所有这些都作为单个虚拟地址空间访问，并由事件驱动的动态自适应计算环境管理。 这种方法的关键方面包括基于未来的同步、动态位置管理、动态资源管理、分层名称空间和活动全局地址空间（AGAS）。 PXFS的总体目标是通过统一命名空间及其管理来消除传统文件系统所强加的编程划分，并最大限度地减少全局同步以支持异步并发。 研究方法是使用PXFS实现Map/Reduce应用框架，并评估其性能和易用性的有效性。该项目在三所主要研究型大学进行，涉及本科生和研究生，博士后，高中教师和他们的学生。 该项目包括一名来自功能基因组学领域的PI，作为领域科学专家，以便将开发工作集中在真实的世界问题上。参与该项目的研究生和博士后在这些领域接受培训，以促进科学家了解大数据问题的两个方面。 该项目吸引了代表性不足的少数群体，旨在激励他们从事计算机科学或基因组学方面的职业。该项目开发的软件是开源的，并使用集成的源代码修订库、wiki和错误跟踪软件系统进行存档，此外还有附带文档的代码发布。

Recent decades have seen the development of computational science where modeling and data analysis are critical to exploration, discovery, and refinement of new innovations in science and engineering. More recently the techniques have been applied to arts, social, political and other fields less traditionally reliant on high performance computing. This innovation has grown out of realization some 20 years ago that I/O (input/output) support for high performance parallel and distributed architectures had lagged behind that of pure computational speed, and further that bring I/O up to speed was both critical, and a rather difficult problem. The core hurdle of contemporary I/O on large HPC machines relates to issues of latency in large parts caused by the deficiencies of the historical I/O model that was relevant when computers were exclusively large, centralized, single processor systems shared by many time-sharing programs. In order to improve I/O on scalability on future hardware architectures novel approaches are required.This project is conducting research on an extension of ParalleX, a new highly innovative parallel execution model. The extension provides a powerful I/O interface that allows researchers to create highly efficient data management, discovery, and analysis codes for Big Data applications. This new extension, known as PXFS, is based on HPX, an implementation of ParalleX based on C++, and OrangeFS, a high performance parallel file system. The research goal driving PXFS is to extend HPX objects into I/O space so that the objects become persistent and storage becomes another class of memory, all accessed as a single virtual address space and managed by an event driven dynamic adaptive computation environment. Critical aspects of this approach include futures-based synchronization, dynamic locality management, dynamic resource management, hierarchical name space, and an active global address space (AGAS). The overall goals of PXFS are to eliminate the division of programming imposed by conventional file system through the unification of name spaces and their management, and to minimize global synchronization in order to support asynchronous concurrency. The research methodology is to implement a Map/Reduce application framework using PXFS and evaluate its effectiveness in both performance and ease of use.This project is conducted at three major research universities involving undergraduate and graduate students, post-docs, and high-school teachers and their students. The project includes a PI from the functional genomics field acting as domain science expert in order to focus the development efforts on real world problems. Graduate students and post-docs involved in the project are trained in these areas to promote scientists who understanding both aspects of Big Data problems. The project engages under represented minorities with the goal to inspire them to pursue a career in computer science or genomics. The software developed by the project is available open-source and archived using an integrated source code revision repository, wiki, and bug tracking software system in addition to code releases with accompanying documentation.