Collaborative Research: Phylanx: Python based Array Processing in HPX

在过去十年中，数据集的可用性和规模显著增加。为了能够在高性能计算（HPC）资源上分析大型数据集，同时最大限度地减少解决方案所需的时间和精力，需要结合静态和运行时信息，以确定应用程序使用的大型数据阵列的最佳数据布局，从而最大限度地减少数据移动。该提案的目标是交付Phylanx，这是一个通用框架，支持各种数据科学、机器学习和面向统计的应用程序。Phylanx的设计使用户？只要运行时系统得到维护，S代码将能够在当前和未来的架构上有效地执行。这将大大减少维护负担，并将提高领域科学家的生产力。Phylanx为从学术界到工业界的技术转移奠定了坚实的基础，通过创建一个行业合作伙伴可以放心依赖的软件层，填补了学术创新与商业应用之间的空白。Phylanx是一个可扩展的，基于阵列的分布式框架，目标是HPC系统，使用HPX，基于动态异步任务的并行运行时系统。HPX暴露的数据流风格功能保证了所有数据依赖的保存，即使是复杂的分布式工作流。该项目克服了现有大数据解决方案（如Hadoop、Spark和Flink）的一些局限性，为用户提供了以下能力：使用Python或C/ c++实现numpy风格的表达式图，优化这些图以获得最佳的数据布局、分布、平铺和最小的通信开销，并在针对分布式HPC系统的运行时解释器上高效地评估这些图。此外，Phylanx在表达式树上使用贪婪子模块技术，为机器学习领域和数据放置问题的最佳性能提供数学上可证明的保证。该平台将提供6个基准测试的实现，这些基准测试是根据它们在文本、图像和图形应用程序中的特定领域而选择的。

The availability and size of data sets has increased significantly over the course of the past decade. To enable the analysis of large data sets on High Performance Computing (HPC) resources while minimizing time- and energy-to-solution requires incorporating static and runtime information to determine the best possible data layout of the large data arrays used by an application to minimize data movement. The goal of this proposal is to deliver Phylanx, a general purpose framework supporting a variety of data science, machine learning, and statistically oriented applications. Phylanx is designed such that a user?s code will be able to perform efficiently on current and future architecture as long as the runtime system is maintained. This greatly reduces the maintenance burden and will increase the productivity of domain scientists. Phylanx lays a solid foundation for technology transfer from academia to industry and fills the gap between academic innovation and commercial application, by creating a software layer that industrial partners can feel confident relying upon. Phylanx is a scalable, array-based and distributed framework targeting HPC systems using the HPX, dynamic asynchronous task-based parallel runtime system. The dataflow-style capabilities exposed by HPX guarantee the preservation of all data-dependencies even for complex distributed workflows. This project overcomes some of the limitations of existing Big Data solutions such as Hadoop, Spark, and Flink by providing users the ability to: implement NumPy-styled expression-graphs using Python or C/C++, optimize these graphs for optimal data layout, distribution, tiling, and minimal communication overheads, and evaluate those graphs with high efficiency on a runtime interpreter targeting distributed HPC systems. Additionally, Phylanx uses greedy sub-modular techniques on the expression tree to provide a mathematically provable guarantee of optimal performance in machine learning domains and in data placement problems. The platform will provide implementations of 6 benchmarks which have been selected for their domain specificity in text, image, and graph applications.