CSR-PSCE, SM: Collaborative Research: VOLPEX: A Framework for Parallel Execution on Volatile Nodes

如今，普通个人电脑被广泛用于大规模的科学计算。BOINC中间件于2004年发布，目前管理着超过50万个志愿PC节点，并为大约30个科学研究项目提供计算能力，包括Rosetta@home、Climateprediction.net和IBM World Community Grid。这种“虚拟集群”的主要吸引力在于计算是“免费的”，因为只需要很少的额外硬件和人力资源。利用这种空闲周期的潜力是巨大的，因为目前世界上10亿台个人电脑中只有不到1%的人参与其中。当前空闲的pc被用于顺序和独立的并行任务，而不是通信并行任务，这是一种常见的HPC范式。该项目的中心目标是在联网的普通pc上实现通信并行应用程序的健壮执行。挑战在于普通桌面是“不稳定的”，也就是说，它们的可用性会根据桌面所有者的操作而突然而频繁地变化。并行应用程序的检查点是容错科学计算的最新技术，对于高故障率的环境是不够的。该项目正在开发VolPEx框架（Volatile node上的并行执行），该框架采用管理冗余作为核心机制，在存在常规故障的情况下实现无缝的向前应用程序进度。规范执行模型由每个流程的两个或多个并发副本组成，其中失败的流程副本根据需要从健康的流程副本中重新生成。为应用程序开发提供了以下通信api。虚拟数据空间：用于任务间异步匿名Put/Get通信的抽象API。独立任务的BOINC编程模型正在用数据空间API进行扩展，以允许任务间通信2。Volpex MPI: MPI的子集实现，具有为在易变节点上执行而定制的通信层。这项研究的验证将包括在Volpex/BOINC下在校园局域网的100个节点和全球1000个节点上执行选定的并行应用程序。将普通pc转换成虚拟集群以运行并行代码的能力将产生广泛的影响。几乎所有的科学家都可以使用高性能计算，而对集群的需求以及购买、操作和维护集群的成本将会减少。

Ordinary PCs are widely employed for large scale scientific computing today. Released in 2004, BOINC middleware is managing over 500,000 volunteered PC nodes and providing computation power to around 30 scientific research projects including Rosetta@home, Climateprediction.net, and IBM World Community Grid. The main attraction of such "virtual clusters" is that computing is "free" as minimal additional hardware and personnel resources are needed. The potential for exploiting such idle cycles is immense since well under 1% of the world's 1 billion PCs are currently participating.Currently idle PCs are exploited for sequential and independent parallel tasks, but not communicating parallel tasks, a common HPC paradigm. The central goal of this project is to achieve robust execution of communicating parallel applications on networked ordinary PCs. The challenge is that ordinary desktops are "volatile", i.e., their availability changes suddenly and frequently based on desktop owner's actions. Checkpointing of parallel applications, the state of the art in fault tolerant scientific computing, is not sufficient for high failure rate environments.This project is developing the VolPEx framework (Parallel Execution on Volatile nodes) that employs managed redundancy as the core mechanism to achieve seamless forward application progress in the presence of routine failures. The canonical execution model consists of two or more concurrent replicas of each process with failed process replicas regenerated on-demand from healthy ones. The following communication APIs are provided for application development.1. Virtual Dataspace: An abstract API for asynchronous anonymous Put/Get communication among tasks. The BOINC programming model of independent tasks is being extended with the dataspace API to allow inter-task communication 2. Volpex MPI: A subset implementation of MPI with a communication layer customized for execution on volatile nodes. The validation of this research will include execution of selected parallel applications on 100s of nodes across campus LANs and 1000s of nodes across the globe under Volpex/BOINC.The ability to transform ordinary PCs into a virtual cluster to run parallel codes will have wide ranging impact. Virtually all scientists will get access to HPC while the need for clusters and the cost of purchasing, operating, and maintaining clusters will diminish.