VAMPIR – Virtualized Non-Functional Memory Properties for Data-Pipeline Scheduling

现代硬件的最新进展可以被认为是HW/SW协同设计的拐点。特别是内存和存储已经经历了前所未有的变化：新技术已经模糊了内存层次结构的传统心理图像;非易失性RAM无疑是质疑这种范式的最突出的例子。然而，非常大的缓存、高带宽存储器（HBM）、非均匀存储器访问（NUMA），甚至远程存储器设计和极快的SSD将NVRAM加入到可用存储器/存储技术的异构组合中。因此，现在的心理图景更像是一个记忆技术池，每一个都定位在一个多维设计空间中。不幸的是，这种大的自由度使得应用程序很难有效地使用这些memors.Within我们设想的VAMPIR项目，我们追求的研究假设，只有一个足够的和创新的OS/App的协同设计可以充分拥抱和利用异构存储器的个人属性。一方面，我们看到对操作系统（OS）的明确需求不仅仅是提供对内存的访问，而是提供具有虚拟地址空间之外的功能的虚拟内存层。例如，操作系统应该透明地提供压缩、加密或增加的容错，直到用于存储器分配的n模冗余。另一方面，我们设想未来的应用程序与操作系统进行更有效的通信，并协商应用程序所需的各个属性。对于这方面，我们专注于数据密集型应用程序的类，部署个人数据流组成的（a）一个网络的依赖数据管道与非阻塞运营商沿着管道和（B）阻塞运营商作为管道之间的连接点。在SPP 2377的上下文中，我们的VAMPIR提案的目标正是基于数据管道优化数据密集型应用程序执行的相互作用，这些应用程序与破坏性内存技术上的虚拟化非功能内存属性有关。我们的核心思想是结合联合收割机（a）操作系统的能力，提供内存分配方面的非功能性要求-在能力以及时间属性-与（B）的机会，数据流水线处理与预定义的时间表的流水线执行和相关的内存要求。我们设想以双向方式调查组合，以允许内存需求和配置的谈判，从而推动OS/App协同设计的信封。

Recent advances in modern hardware can be considered an inflection point for HW/SW co-design. Especially memory and storage has seen an unprecedented change: Novel techniques have blurred the traditional mental picture of the memory hierarchy; non-volatile RAM is certainly the most prominent example to question this paradigm. However, very large caches, High-Bandwidth-Memory (HBM), Non-Uniform Memory Access (NUMA), or even remote-memory designs and extremely fast SSDs join NVRAM in the heterogeneous portfolio of available memory/storage techniques. Consequently, the mental picture is now rather that of a pool of memory techniques, each positioned in a multidimensional design space. Unfortunately, this large degree of freedom makes it difficult for applications to efficiently and effectively use these memories.Within our envisioned VAMPIR project, we pursue the research hypothesis that only an adequate and innovative OS/App co-design can fully embrace and exploit the individual properties of heterogeneous memories. On the one hand, we see a clear demand for an operating system (OS) to offer more than just access to memory, but to provide a virtualized memory layer with features beyond virtual address spaces. For example, the OS should transparently provide compression, encryption, or increased fault-tolerance up to n-modular redundancy for memory allocations. On the other hand, we envision future applications to communicate much more descriptively with the OS and negotiate individual properties required by the application. For this aspect, we focus on the class of data-intensive applications, which deploy individual data streams consisting of (a) a network of dependent data pipelines with non-blocking operators along a pipeline and (b) blocking operators as connection points between pipelines. Within the context of the SPP 2377, our VAMPIR proposal targets exactly the interplay of optimizing the execution of data-intensive applications based on data pipelines with respect to virtualized non-functional memory properties on disruptive memory technologies. Our core idea is to combine (a) the capability of an OS providing memory allocations with respect to non-functional requirements – in terms of capabilities as well as temporal properties – with (b) the opportunity of data-pipeline processing with a pre-defined schedule of pipeline executions and dependent memory requirements. We envision to investigate the combination in a bi-directional manner to allow negotiation of memory requirements and provisioning, thus pushing the envelope in OS/App co-design.