OAC Core: A Scalable and Deployable Container Orchestration Cyber Infrastructure Toolkit for Deploying Big Data Analytics Applications in Public Cloud

如今，许多大数据分析应用程序(例如欺诈检测、社交数据分析、教育、气候建模、流行病学和金融)需要处理来自地理分布位置的海量数据集。一个新兴趋势是在公共云中托管这些大数据分析应用。它们可以打包在轻量级隔离执行环境(容器)中运行，并部署在从公共云提供商那里租用的计算资源上，这些计算资源可以无缝更新和扩展。然而，复杂的容器间关联和硬件资源的异构性给管理公共云中的这些大数据分析应用带来了巨大的挑战。该项目支持在公共云中轻松部署集装化大数据分析应用程序，并为云提供商提供洞察力，以便更好地针对当前和未来的大数据工作负载调整其系统。该项目的目标是开发可扩展和可部署的网络基础设施(CI)容器协调工具包，用于在最先进的公共多/混合云中的异类节点上部署大量集装化大数据分析应用程序。该项目跨越了三个互补的推力：(I)实现了一个新颖的黑盒轻量级工具，它通过分层聚类和共现分析以非侵入性的方式检测集装箱大数据分析应用程序的容器间相关性；(Ii)实现了一个新型的可扩展容器调度器，它以相关性感知的方式在公共云中的不同节点上部署集装箱大数据分析应用程序；(Iii)该系统在开源容器协调工具上实现，并通过在基于实验室的原型和实际的真实世界数据中心上进行实验进行验证。除技术贡献外，该项目还包括各种教育和外联活动。研究结果被整合到本科和研究生系统课程中。最后，在这个项目中开发的工具包、源代码、数据集和课程材料被记录下来并开源。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Today many big data analytics applications (e.g., fraud detection, social data analytics, education, climate modeling, epidemiology, and finance) need to process enormous datasets from geographically distributed locations. An emerging trend is to host these big data analytics applications in the public cloud. They can be packaged to run in a lightweight isolated execution environment (containers) and deployed on computing resources rented from public cloud providers, which can be updated and scaled seamlessly. However, the complex inter-container correlations and the heterogeneity of hardware resources pose significant challenges in managing these big data analytics applications in the public cloud. This project enables the easy deployment of containerized big data analytics applications in the public cloud and provides cloud providers with insights to better tune their systems for current and future big data workloads.The goal of this project is to develop a scalable and deployable cyber infrastructure (CI) container orchestration toolkit for deploying large numbers of containerized big data analytics applications on heterogeneous nodes in state-of-the-art public multi/hybrid-cloud. This project spans three complementary thrusts: (i) a novel “black-box” lightweight tool is implemented, which detects inter-container correlations for containerized big data analytics applications in a non-intrusive manner via hierarchical clustering and co-occurrence analysis; (ii) a novel scalable container scheduler is implemented, which deploys containerized big data analytics applications on heterogeneous nodes in the public cloud in a correlation-aware manner; and (iii) the system is implemented on open-source container orchestration tools and validated by subjecting it to experimentation on both the lab-based prototype and the practical, real-world data centers. In addition to its technical contributions, this project involves various educational and outreach activities as well. The results of the research are integrated into the undergraduate and graduate systems courses. Finally, the toolkit, source code, datasets, and course materials developed in this project are documented and open-sourced.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.