Managing the Performance of Big Data Analytics on Heterogeneous Infrastructures

对于一系列跨越基础科学和应用科学的重大科学计算挑战，在大规模系统（如内部或外部云、集群甚至分布式公共资源（“人群计算”））上部署大数据分析（Big Data Analytics，以下简称BDAs）需要提供可预测性能和使用成本的保证。然而，目前这是不可能的，因为科学界在建模和分析层面都缺乏技术，无法确定bda的关键特征及其对性能的影响。也很少有数据或模拟可以解决系统操作和基础设施在定义总体性能中的作用。***本研究的总体目标是通过更深入地了解如何优化运行在大型基础设施甚至混合基础设施上的系统上的bda的部署来填补这一空白，以便在考虑运行成本的同时实现最佳性能。总的来说，我们将使用大数据分析和建模结果的新颖组合来实现这一目标。***我们的研究将涉及bda在工作负载、数据和资源维度方面的建模，以及bda在拟议建模方面的分析。我们将探索部署bda的替代系统，从私有集群到云计算和群体计算。我们将开发一种用于部署在云、集群和人群混合系统中的bda的性能预测方法。这将通过创建多智能体利用模型和用数值和解析方法分析计算环境来实现。我们将使用这些预测来创建针对系统利用率的成本限制的性能优化方案。这些方案将通过调整（即扩展或混合）系统来适应执行。在我们的研究中，我们将对bda的一系列科学和商业应用以及各种系统进行观察和实验，这些系统存在定性和定量差异。这将使我们有机会创建适用于各种bda环境的解决方案，但也可以根据bda的特征和系统的特征，挖掘基于分析、预测和部署的通用指南的性能优化的局限性。

For a range of major scientific computing challenges that span fundamental and applied science, the deployment of Big Data Analytics (hereafter BDAs) on a large-scale system, such as an internal or external cloud, a cluster or even distributed public resources (crowd computing“), needs to be offered with guarantees of predictable performance and utilization cost. Currently, however, this is not possible, because scientific communities lack the technology, both at the level of modelling and analytics, that identifies the key characteristics of BDAs and their impact on performance. There is also little data or simulations available that address the role of the system operation and infrastructure in defining overall performance. ***The overall goal of the proposed research is to fill this gap by producing a deeper understanding of how to optimize the deployment of BDAs that run on systems operating on large infrastructures, and even hybrid infrastructures, in order to achieve optimal performance, while taking into account running costs. Overall, we will achieve this using a novel combination of big data analytics and modeling results.***Our research will involve the modeling of BDAs with respect to dimensions of workload, data and resources, and profiling of BDAs with respect to the proposed modeling. We will explore alternative systems for the deployment of BDAs, ranging from private clusters, to cloud computing and crowd computing. We will develop a methodology for the performance prediction of BDAs deployed in hybrid systems of cloud, cluster and crowd. This will be achieved with the creation of a multi-agent utilization model and the analysis of the computing environment with numerical and analytical methods. We will employ the predictions to create schemes for performance optimization with respect to cost limitations for system utilization. The schemes will accommodate execution by adapting, i.e. expanding or hybridizing, the system. In our research we will observe and experiment with a range of scientific and business applications of BDAs, as well as a variety of systems, which present qualitative and quantitative differences. These will give us the opportunity to create solutions that are applicable to wide range of BDAs environments, but also mine the limitations of performance optimization based on generic guidelines for profiling, prediction and deployment, with respect to the characteristics of BDAs and the characteristics of the system.