Managing the Performance of Big Data Analytics on Heterogeneous Infrastructures

管理异构基础设施上的大数据分析性能

• 批准号: RGPIN-2018-04332
• 负责人: Kantere, Vasiliki
• 金额: $ 1.68万
• 依托单位: University of Ottawa
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2019
• 资助国家: 加拿大
• 起止时间: 2019-01-01 至 2020-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=679322
• 关键词: Managing; Performance; Big; Data; Analytics

For a range of major scientific computing challenges that span fundamental and applied science, the deployment of Big Data Analytics (hereafter BDAs) on a large-scale system, such as an internal or external cloud, a cluster or even distributed public resources (crowd computing“), needs to be offered with guarantees of predictable performance and utilization cost. Currently, however, this is not possible, because scientific communities lack the technology, both at the level of modelling and analytics, that identifies the key characteristics of BDAs and their impact on performance. There is also little data or simulations available that address the role of the system operation and infrastructure in defining overall performance. ***The overall goal of the proposed research is to fill this gap by producing a deeper understanding of how to optimize the deployment of BDAs that run on systems operating on large infrastructures, and even hybrid infrastructures, in order to achieve optimal performance, while taking into account running costs. Overall, we will achieve this using a novel combination of big data analytics and modeling results.***Our research will involve the modeling of BDAs with respect to dimensions of workload, data and resources, and profiling of BDAs with respect to the proposed modeling. We will explore alternative systems for the deployment of BDAs, ranging from private clusters, to cloud computing and crowd computing. We will develop a methodology for the performance prediction of BDAs deployed in hybrid systems of cloud, cluster and crowd. This will be achieved with the creation of a multi-agent utilization model and the analysis of the computing environment with numerical and analytical methods. We will employ the predictions to create schemes for performance optimization with respect to cost limitations for system utilization. The schemes will accommodate execution by adapting, i.e. expanding or hybridizing, the system. In our research we will observe and experiment with a range of scientific and business applications of BDAs, as well as a variety of systems, which present qualitative and quantitative differences. These will give us the opportunity to create solutions that are applicable to wide range of BDAs environments, but also mine the limitations of performance optimization based on generic guidelines for profiling, prediction and deployment, with respect to the characteristics of BDAs and the characteristics of the system.

对于一系列跨越基础科学和应用科学的重大科学计算挑战，在大规模系统上部署大数据分析（以下简称BDA），例如内部或外部云，集群甚至分布式公共资源（“人群计算”），需要提供可预测性能和利用成本的保证。然而，目前这是不可能的，因为科学界在建模和分析方面都缺乏技术，无法确定业务发展分析的关键特征及其对业绩的影响。也有一些数据或模拟，解决系统操作和基础设施在定义整体性能的作用。* 拟议研究的总体目标是通过深入了解如何优化在大型基础设施甚至混合基础设施上运行的系统上运行的BDA的部署来填补这一空白，以便在考虑运行成本的同时实现最佳性能。 总的来说，我们将使用大数据分析和建模结果的新颖组合来实现这一目标。我们的研究将涉及工作负载，数据和资源方面的BDA的建模，以及对BDA的分析。我们将探索部署BDA的替代系统，从私有集群到云计算和群计算。我们将开发一种方法来预测部署在云，集群和人群的混合系统中的BDA的性能。这将通过建立一个多代理人利用模式和用数字和分析方法分析计算环境来实现。我们将采用的预测，以创建性能优化方案的系统利用率的成本限制。这些计划将通过调整，即扩大或混合该系统来适应执行。在我们的研究中，我们将观察和实验BDA的一系列科学和商业应用，以及各种系统，这些系统存在定性和定量差异。这将使我们有机会创建适用于各种BDA环境的解决方案，但也可以根据BDA的特性和系统的特性，根据分析、预测和部署的通用指南挖掘性能优化的局限性。