CAREER: Transparent, Interactive Desktop Parallel Computing for Scientific Data Processing

桌面计算在科学探索中仍然是不可或缺的，很大程度上是因为它为人们提供了人机交互的设备和交互式工作执行的环境。事实上，超级计算机和集群的激增已经推动了对更高效的桌面处理的需求，以协助工具验证、数据分析和可视化中的高端计算。然而，随着数据量和任务复杂性的快速增长，单个工作站越来越难以满足交互式科学数据处理的需求。这种交互处理的成本不断增加，阻碍了端到端科学计算工作流程的生产力。该项目将开发一种新型的桌面并行计算框架，以加速通常在桌面机器上执行的科学数据处理任务。它将允许用户保持桌面处理的交互性和便利性，而不需要显式的资源请求或等待批处理执行，同时他们的计算通过在局域网中聚合空闲的计算和存储资源而无缝地加速。该框架包含了本研究将要开发的几个紧密耦合的创新技术：基于关系代数的数据处理语义规范，在此基础上设计了实现自动灵活程序并行化的应用接口；集成了计算资源聚合和存储资源聚合，在此基础上开发了桌面并行计算的并行I/O；利用客户端工作站中心角色的非对称任务调度，以保证交互式执行、更好的容错能力和多种自配置机会；以及基于影响基准测试和实时工作负载监控的定量和显式性能影响控制，在存在侵略性和持久性资源窃取的情况下，保护资源提供者本地工作负载的性能。提出的框架具有三个关键属性，使其与现有的并行执行平台显著不同：交互性（无论外部资源是否可用都立即执行）、透明性（对应用程序开发人员和用户隐藏外部资源的可用性和波动）以及自定义的性能影响控制（根据从单个资源所有者的本机工作负载测量的资源使用情况来限制并行作业资源消耗）。我们的研究朝着适合异构和机会环境的新的并行计算范式迈出了最初的一步。使用这个新范例，用户不必指定所需资源的数量，也不必等待这些资源变得可用。

Desktop computing remains indispensable in scientific exploration, largely because it provides people with devices for human interaction and environments for interactive job execution. In fact, the proliferation of supercomputers and clusters has been driving the need for more efficient desktop processing, to assist high-end computing in tool validation, data analysis, and visualization. However, with the rapidly growing data volume and task complexity, it is increasingly hard for individual workstations to meet the demands of interactive scientific data processing. The increasing cost of such interactive processing is hindering the productivity of end-to-end scientific computing workflows. The project will develop a novel desktop parallel computing framework to speed up scientific data processing tasks routinely executed on desktop machines. It will allow users to preserve the interactiveness and convenience of desktop processing, without explicit resource request or waiting for batch execution, while their computation is seamlessly accelerated by aggregating idle computing and storage resources in local-area networks. This framework comprises several closely coupled innovative techniques to be developed in this research: data processing semantics specification based on relational algebra, on top of which we design application interfaces for automatic and flexible program parallelization; integrated computing resource aggregation and storage resource aggregation, on top of which we developparallel I/O for desktop parallel computing; asymmetric task scheduling that exploits the central role of the client workstation, for guaranteed interactive execution, better fault tolerance, and diverse self-configuration opportunities; and quantitative and explicit performance impact control based on impact benchmarking and real-time workload monitoring that protects the performance of resource donors native workloads, in the presence of aggressive and persistent resource stealing. The proposed framework possesses three key properties that distinguish itself significantly from existing parallel execution platforms: interactiveness (immediate execution regardless of the availability of external resources), transparency (hiding the availability of and fluctuation in external resources from both application developers and users), and customized performance impact control (throttling a parallel jobs resource consumption according to the resource usage measured from the native workload of individual resource owners). Our research takes the initial steps towards a new parallel computing paradigm suitable for heterogeneous and opportunistic environments. With this new paradigm, users do not have to specify the amount of resources needed, nor do they have to wait for such resources to become available.