Managing the Performance of Big Data Analytics on Heterogeneous Infrastructures

对于跨越基础科学和应用科学的一系列重大科学计算挑战，在大型系统上部署大数据分析(以下简称BDA)需要提供可预测的性能和使用成本保证，例如内部或外部云、集群甚至分布式公共资源(“众筹计算”)。然而，目前这是不可能的，因为科学界在建模和分析方面缺乏技术，无法确定生物多样性指标的主要特征及其对绩效的影响。也几乎没有可用的数据或模拟来说明系统操作和基础设施在定义总体性能方面的作用。*拟议研究的总体目标是通过更深入地了解如何优化在大型基础设施甚至混合基础设施上运行的系统上运行的BDA的部署，以实现最佳性能，同时考虑运行成本，从而填补这一空白。总体而言，我们将使用大数据分析和建模结果的新组合来实现这一点。*我们的研究将涉及针对工作负载、数据和资源的维度对BDA进行建模，以及针对建议的建模对BDA进行配置。我们将探索部署BDA的替代系统，范围从私有集群到云计算和众筹计算。我们将制定一种方法来预测部署在云、集群和人群混合系统中的BDA的性能。这将通过建立多代理利用模型和用数值和分析方法分析计算环境来实现。我们将使用这些预测来创建关于系统利用率的成本限制的性能优化方案。这些计划将通过调整、即扩展或混合该系统来适应执行。在我们的研究中，我们将观察和试验BDAS的一系列科学和商业应用，以及呈现定性和定量差异的各种系统。这些将使我们有机会创建适用于各种BDAS环境的解决方案，但也将根据BDAS的特点和系统的特点，根据分析、预测和部署的一般指导方针，挖掘性能优化的局限性。

For a range of major scientific computing challenges that span fundamental and applied science, the deployment of Big Data Analytics (hereafter BDAs) on a large-scale system, such as an internal or external cloud, a cluster or even distributed public resources („crowd computing“), needs to be offered with guarantees of predictable performance and utilization cost. Currently, however, this is not possible, because scientific communities lack the technology, both at the level of modelling and analytics, that identifies the key characteristics of BDAs and their impact on performance. There is also little data or simulations available that address the role of the system operation and infrastructure in defining overall performance. ***The overall goal of the proposed research is to fill this gap by producing a deeper understanding of how to optimize the deployment of BDAs that run on systems operating on large infrastructures, and even hybrid infrastructures, in order to achieve optimal performance, while taking into account running costs. Overall, we will achieve this using a novel combination of big data analytics and modeling results.***Our research will involve the modeling of BDAs with respect to dimensions of workload, data and resources, and profiling of BDAs with respect to the proposed modeling. We will explore alternative systems for the deployment of BDAs, ranging from private clusters, to cloud computing and crowd computing. We will develop a methodology for the performance prediction of BDAs deployed in hybrid systems of cloud, cluster and crowd. This will be achieved with the creation of a multi-agent utilization model and the analysis of the computing environment with numerical and analytical methods. We will employ the predictions to create schemes for performance optimization with respect to cost limitations for system utilization. The schemes will accommodate execution by adapting, i.e. expanding or hybridizing, the system. In our research we will observe and experiment with a range of scientific and business applications of BDAs, as well as a variety of systems, which present qualitative and quantitative differences. These will give us the opportunity to create solutions that are applicable to wide range of BDAs environments, but also mine the limitations of performance optimization based on generic guidelines for profiling, prediction and deployment, with respect to the characteristics of BDAs and the characteristics of the system.