DECICE (HORIZON-CL4-2022-DATA-01)

云计算行业在过去十年中大规模增长，并出现了新的应用领域。某些区域需要专门的硬件，这些硬件需要放置在靠近用户的位置。超低延迟、安全性和位置感知等用户需求在智能城市、工业自动化和数据分析等领域变得越来越普遍。现代云应用程序也变得更加复杂，因为它们通常在分布式计算机系统上运行，并被拆分为必须以高可用性运行的组件。将这些不同的系统统一到集中控制的计算集群中，并在它们之间提供复杂的调度决策是该领域的两个主要挑战。由云和边缘节点组成的集群的调度决策必须考虑独特的特征，例如节点和网络容量的可变性。编排大型集群的常见解决方案是Kubernetes，然而，它是为可靠的同构集群而设计的。许多应用程序和扩展可用于Kubernetes。不幸的是，它们都没有考虑到性能和能源的优化，或者解决数据和作业本地化问题。在DECICE中，我们开发了一个开放和可移植的云管理框架，通过将作业映射到异构系统环境中最合适的资源来自动和自适应地优化应用程序。通过利用整体监控，我们构建了一个反映原始系统的数字孪生系统。人工智能调度器决定作业和数据的放置，并进行作业重新调度以适应系统的变化。提供了虚拟训练环境，其生成用于ML模型的训练和假设场景的探索的测试数据。可移植框架集成到Kubenetes生态系统中，并使用真实世界异构系统上的相关用例进行验证。

The cloud computing industry has grown massively over the last decade and with that new areas of application have arisen. Some areas require specialized hardware, which needs to be placed in locations close to the user. User requirements such as ultra-low latency, security and location awareness are becoming more and more common, for example, in Smart Cities, industrial automation and data analytics. Modern cloud applications have also become more complex as they usually run on a distributed computer system, split up into components that must run with high availability. Unifying such diverse systems into centrally controlled compute clusters and providing sophisticated scheduling decisions across them are two major challenges in this field. Scheduling decisions for a cluster consisting of cloud and edge nodes must consider unique characteristics such as variability in node and network capacity. The common solution for orchestrating large clusters is Kubernetes, however, it is designed for reliable homogeneous clusters. Many applications and extensions are available for Kubernetes. Unfortunately, none of them accounts for optimization of both performance and energy or addresses data and job locality. In DECICE, we develop an open and portable cloud management framework for automatic and adaptive optimization of applications by mapping jobs to the most suitable resources in a heterogeneous system landscape. By utilizing holistic monitoring, we construct a digital twin of the system that reflects on the original system. An AI-scheduler makes decisions on placement of job and data as well as conducting job rescheduling to adjust to system changes. A virtual training environment is provided that generates test data for training of ML-models and the exploration of what-if scenarios. The portable framework is integrated into the Kubenetes ecosystem and validated using relevant use cases on real-world heterogeneous systems.