VAMPIR – Virtualized Non-Functional Memory Properties for Data-Pipeline Scheduling

点击翻译按钮获取中文摘要

Recent advances in modern hardware can be considered an inflection point for HW/SW co-design. Especially memory and storage has seen an unprecedented change: Novel techniques have blurred the traditional mental picture of the memory hierarchy; non-volatile RAM is certainly the most prominent example to question this paradigm. However, very large caches, High-Bandwidth-Memory (HBM), Non-Uniform Memory Access (NUMA), or even remote-memory designs and extremely fast SSDs join NVRAM in the heterogeneous portfolio of available memory/storage techniques. Consequently, the mental picture is now rather that of a pool of memory techniques, each positioned in a multidimensional design space. Unfortunately, this large degree of freedom makes it difficult for applications to efficiently and effectively use these memories.Within our envisioned VAMPIR project, we pursue the research hypothesis that only an adequate and innovative OS/App co-design can fully embrace and exploit the individual properties of heterogeneous memories. On the one hand, we see a clear demand for an operating system (OS) to offer more than just access to memory, but to provide a virtualized memory layer with features beyond virtual address spaces. For example, the OS should transparently provide compression, encryption, or increased fault-tolerance up to n-modular redundancy for memory allocations. On the other hand, we envision future applications to communicate much more descriptively with the OS and negotiate individual properties required by the application. For this aspect, we focus on the class of data-intensive applications, which deploy individual data streams consisting of (a) a network of dependent data pipelines with non-blocking operators along a pipeline and (b) blocking operators as connection points between pipelines. Within the context of the SPP 2377, our VAMPIR proposal targets exactly the interplay of optimizing the execution of data-intensive applications based on data pipelines with respect to virtualized non-functional memory properties on disruptive memory technologies. Our core idea is to combine (a) the capability of an OS providing memory allocations with respect to non-functional requirements – in terms of capabilities as well as temporal properties – with (b) the opportunity of data-pipeline processing with a pre-defined schedule of pipeline executions and dependent memory requirements. We envision to investigate the combination in a bi-directional manner to allow negotiation of memory requirements and provisioning, thus pushing the envelope in OS/App co-design.