CAREER: Transparent, Interactive Desktop Parallel Computing for Scientific Data Processing

桌面计算在科学探索中仍然是不可或缺的，这主要是因为它为人们提供了人机交互的设备和交互式作业执行的环境。事实上，超级计算机和集群的激增一直在推动对更高效的桌面处理的需求，以协助工具验证，数据分析和可视化的高端计算。然而，随着数据量和任务复杂性的快速增长，单个工作站越来越难以满足交互式科学数据处理的需求。这种交互式处理的成本不断增加，阻碍了端到端科学计算工作流程的生产力。 该项目将开发一种新的桌面并行计算框架，以加快在桌面机器上常规执行的科学数据处理任务。它将允许用户保留桌面处理的交互性和便利性，而无需显式的资源请求或等待批处理执行，同时通过聚合局域网中的空闲计算和存储资源来无缝加速他们的计算。该框架包含了几个紧密结合的创新技术：基于关系代数的数据处理语义规范，在此基础上设计了自动灵活的程序并行化应用接口，集成了计算资源聚合和存储资源聚合，在此基础上开发了桌面并行计算的并行I/O;非对称任务调度，其利用客户端工作站的中心作用，以保证交互式执行、更好的容错性和多样化的自配置机会;以及基于影响基准和真实的-时间工作负载监控，在存在侵略性和持续性资源窃取的情况下保护资源捐赠者本机工作负载的性能。 所提出的框架具有三个关键属性，使其与现有的并行执行平台显着不同：交互性（不论是否有外部资源，都立即执行）、透明度（对应用程序开发人员和用户隐藏外部资源的可用性和波动），和定制的性能影响控制（根据从各个资源所有者的本机工作负载测量的资源使用来节流并行作业资源消耗）。我们的研究采取了初步的步骤，一个新的并行计算模式，适合异构和机会主义的环境。有了这个新的范例，用户不必指定所需的资源量，也不必等待这些资源变得可用。

Desktop computing remains indispensable in scientific exploration, largely because it provides people with devices for human interaction and environments for interactive job execution. In fact, the proliferation of supercomputers and clusters has been driving the need for more efficient desktop processing, to assist high-end computing in tool validation, data analysis, and visualization. However, with the rapidly growing data volume and task complexity, it is increasingly hard for individual workstations to meet the demands of interactive scientific data processing. The increasing cost of such interactive processing is hindering the productivity of end-to-end scientific computing workflows. The project will develop a novel desktop parallel computing framework to speed up scientific data processing tasks routinely executed on desktop machines. It will allow users to preserve the interactiveness and convenience of desktop processing, without explicit resource request or waiting for batch execution, while their computation is seamlessly accelerated by aggregating idle computing and storage resources in local-area networks. This framework comprises several closely coupled innovative techniques to be developed in this research: data processing semantics specification based on relational algebra, on top of which we design application interfaces for automatic and flexible program parallelization; integrated computing resource aggregation and storage resource aggregation, on top of which we developparallel I/O for desktop parallel computing; asymmetric task scheduling that exploits the central role of the client workstation, for guaranteed interactive execution, better fault tolerance, and diverse self-configuration opportunities; and quantitative and explicit performance impact control based on impact benchmarking and real-time workload monitoring that protects the performance of resource donors native workloads, in the presence of aggressive and persistent resource stealing. The proposed framework possesses three key properties that distinguish itself significantly from existing parallel execution platforms: interactiveness (immediate execution regardless of the availability of external resources), transparency (hiding the availability of and fluctuation in external resources from both application developers and users), and customized performance impact control (throttling a parallel jobs resource consumption according to the resource usage measured from the native workload of individual resource owners). Our research takes the initial steps towards a new parallel computing paradigm suitable for heterogeneous and opportunistic environments. With this new paradigm, users do not have to specify the amount of resources needed, nor do they have to wait for such resources to become available.