Intelligent dynamic scheduling in clusters and computational grids

由于其高度分布式的性质，计算网格在加拿大的研究领域扮演着重要的角色，并得到多个组织和资助机构的支持，如GridCanada、C3、CANARIE和CFI-例如，通过Sharcnet集群网络，服务于高度多样化的应用领域，如化学、工程和经济。计算网格提供了加入和共享HPC资源的选项，用于解决超出本地站点能力范围的大规模问题，或者用于在站点之间平衡本地工作负载中的热点。对于大规模作业的调度，可以选择粗粒度工作流或同时资源分配，本方案针对的是后者。因此，很难在不同的网站上找到匹配的日程安排时间。此外，调度器必须处理异构性、现代多核CPU(如AMD Opteron)中可用的新多处理功能以及数据挖掘等新的I/O密集型应用程序。这些趋势使得作业调度方法变得必要，这种方法是灵活和动态的，以提高机器利用率并在不损害本地工作负载的情况下找到同步时隙。这项研究以自适应资源分配和分时作业执行的形式解决了这种动态方法。其中一些方法还需要中间件来使应用程序具有适应性。总之，在网格、作业和应用程序级别都需要创新的解决方案。该研究提出了自适应作业调度、多核处理器上的特殊分时作业调度、将这些方法集成到网格调度中、具有经济模型的特殊网格调度器以及具有高质量数据分发的自适应中间件的解决方案，用于一类重要的通信关键型数值应用。解决方案实际上将得到工具环境的支持，理论上将得到高级成本建模的支持。此外，解决方案将被设计为可配置为具有应用程序和资源感知能力，并将提取适合范围。

Computational grids play a major role in the Canadian research landscape due its highly distributed nature and are supported by several organizations and funding agencies like GridCanada, C3, Canarie, and CFI - e.g., via the Sharcnet network of clusters, serving highly diverse application areas like chemistry, engineering, and economics. Computational grids provide the option to join and share HPC resources for tackling large-scale problems that go beyond the local site's capacities or for balancing hot spots in local workloads among sites. For scheduling of large-scale jobs, either coarse-grain workflow or simultaneous resource allocation may be chosen, of which this proposal addresses the latter. Finding matching scheduling slots on different sites, then, is difficult. Furthermore, schedulers have to deal with heterogeneity, new multi-processing features available in modern multi-core CPUs like the AMD Opteron, and new I/O-intensive applications like datamining. These trens make job-scheduling approaches necessary that are flexible and dynamic to improve machine utilization and to find simultaneous time slots without harming local workload. This research addresses such dynamic approaches in the form of adaptive resource allocation and time-shared job execution. Some of these approaches also require middleware to make the applications adaptable. Altogether, innovative solutions are required at the grid, job, and application level. This research proposes solutions for adaptive job scheduling, special time-shared job scheduling on multi-core CPUs, an integration of these approaches into grid scheduling, special grid schedulers with economic models, and adaptive middleware with high-quality data distribution for an important class of communication-critical numeric applications. The solutions will be practically supported by a tool environment and theoretically by advanced cost modeling. Furthermore, the solutions will be designed configurable to be application- and resource-aware, and suitability ranges will be extracted.