OAC Core: A Scalable and Deployable Container Orchestration Cyber Infrastructure Toolkit for Deploying Big Data Analytics Applications in Public Cloud

OAC Core：用于在公共云中部署大数据分析应用程序的可扩展和可部署的容器编排网络基础设施工具包

• 批准号: 2313738
• 负责人: Liting Hu
• 金额: $ 32.43万
• 依托单位: University of California-Santa Cruz
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-02-01 至 2025-12-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2313738&HistoricalAwards=false
• 关键词: OAC; Core; Scalable; Deployable; Container

Today many big data analytics applications (e.g., fraud detection, social data analytics, education, climate modeling, epidemiology, and finance) need to process enormous datasets from geographically distributed locations. An emerging trend is to host these big data analytics applications in the public cloud. They can be packaged to run in a lightweight isolated execution environment (containers) and deployed on computing resources rented from public cloud providers, which can be updated and scaled seamlessly. However, the complex inter-container correlations and the heterogeneity of hardware resources pose significant challenges in managing these big data analytics applications in the public cloud. This project enables the easy deployment of containerized big data analytics applications in the public cloud and provides cloud providers with insights to better tune their systems for current and future big data workloads.The goal of this project is to develop a scalable and deployable cyber infrastructure (CI) container orchestration toolkit for deploying large numbers of containerized big data analytics applications on heterogeneous nodes in state-of-the-art public multi/hybrid-cloud. This project spans three complementary thrusts: (i) a novel “black-box” lightweight tool is implemented, which detects inter-container correlations for containerized big data analytics applications in a non-intrusive manner via hierarchical clustering and co-occurrence analysis; (ii) a novel scalable container scheduler is implemented, which deploys containerized big data analytics applications on heterogeneous nodes in the public cloud in a correlation-aware manner; and (iii) the system is implemented on open-source container orchestration tools and validated by subjecting it to experimentation on both the lab-based prototype and the practical, real-world data centers. In addition to its technical contributions, this project involves various educational and outreach activities as well. The results of the research are integrated into the undergraduate and graduate systems courses. Finally, the toolkit, source code, datasets, and course materials developed in this project are documented and open-sourced.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

如今，许多大数据分析应用程序(例如欺诈检测、社交数据分析、教育、气候建模、流行病学和金融)需要处理来自地理分布位置的海量数据集。一个新兴趋势是在公共云中托管这些大数据分析应用。它们可以打包在轻量级隔离执行环境(容器)中运行，并部署在从公共云提供商那里租用的计算资源上，这些计算资源可以无缝更新和扩展。然而，复杂的容器间关联和硬件资源的异构性给管理公共云中的这些大数据分析应用带来了巨大的挑战。该项目支持在公共云中轻松部署集装化大数据分析应用程序，并为云提供商提供洞察力，以便更好地针对当前和未来的大数据工作负载调整其系统。该项目的目标是开发可扩展和可部署的网络基础设施(CI)容器协调工具包，用于在最先进的公共多/混合云中的异类节点上部署大量集装化大数据分析应用程序。该项目跨越了三个互补的推力：(I)实现了一个新颖的黑盒轻量级工具，它通过分层聚类和共现分析以非侵入性的方式检测集装箱大数据分析应用程序的容器间相关性；(Ii)实现了一个新型的可扩展容器调度器，它以相关性感知的方式在公共云中的不同节点上部署集装箱大数据分析应用程序；(Iii)该系统在开源容器协调工具上实现，并通过在基于实验室的原型和实际的真实世界数据中心上进行实验进行验证。除技术贡献外，该项目还包括各种教育和外联活动。研究结果被整合到本科和研究生系统课程中。最后，在这个项目中开发的工具包、源代码、数据集和课程材料被记录下来并开源。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Adaptive Fragment-Based Parallel State Recovery for Stream Processing Systems

• DOI: 10.1109/tpds.2023.3251997
• 发表时间: 2023-08
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Hailu Xu;Pinchao Liu;Sarker Tanzir Ahmed;Dilma Da Silva;Liting Hu

Achieving Online and Scalable Information Integrity by Harnessing Social Spam Correlations

• DOI: 10.1109/access.2023.3236604
• 发表时间: 2023
• 期刊: IEEE Access
• 影响因子: 3.9
• 作者: Hailu Xu;Pinchao Liu;Boyuan Guan;Qingyang Wang;Dilma Da Silva;Liting Hu

XUnified: A Framework for Guiding Optimal Use of GPU Unified Memory

XUnified：指导 GPU 统一内存优化使用的框架

• DOI: 10.1109/access.2022.3196008
• 发表时间: 2022
• 期刊: IEEE Access
• 影响因子: 3.9
• 作者: Xu, Hailu;Lin, Pei-Hung;Emani, Murali;Hu, Liting;Liao, Chunhua
• 通讯作者: Liao, Chunhua

Sora: A Latency Sensitive Approach for Microservice Soft Resource Adaption

Sora：一种延迟敏感的微服务软资源适配方法

• 发表时间: 2023
• 期刊: 24th ACM/IFIP International Middleware Conference
• 作者: Liu, Jianshu;Wang, Qingyang;Liu, Jianshu;Zhang, Shungeng;Hu, Liting;Silva, Dilma Da
• 通讯作者: Silva, Dilma Da