ADA-FS - Advanced Data Placement via Ad-hoc File Systems at Extreme Scales

ADA-FS - 通过极端规模的临时文件系统进行高级数据放置

• 批准号: 279334509
• 负责人: Professor Dr.-Ing. André Brinkmann
• 金额: --
• 依托单位: Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH)
• 依托单位国家: 德国
• 项目类别: Priority Programmes
• 财政年份: 2015
• 资助国家: 德国
• 起止时间: 2014-12-31 至 2019-12-31
• 项目状态: 已结题
• 来源: https://gepris.dfg.de/gepris/projekt/279334509?language=en
• 关键词: ADA; FS; Advanced; Data; Placement

The proposed ADA-FS project aims at improving the I/O performance for extreme-scale parallel applications. It will introduce three novel components to the system software. First, it proposes the use of an ad-hoc overlay file system concept. Second, a central I/O planner component is envisioned. The third component integrates a concept for application monitoring, with emphasis on its resource usage and I/O behavior. The ad-hoc overlay file system will be deployed as a separate instance for every parallel application on the same compute nodes that the application is running on. This file system will use traditional and novel system components such as RAM, local SSDs, or NVRAM as data caches for the I/O subsystem provided that these are not in use by the particular application. This approach will increase the effective I/O bandwidth and reduce the latency of I/O operations. Parallel applications will require no modifications to make use of this ad-hoc file system. The central I/O planner will coordinate all data transfers to and from all instances of the overlay file system. The planner will prevent a random mix of I/O requests in which no application can anticipate how much of the total I/O bandwidth is available at any given point in time in the globally shared I/O infrastructure. Instead, the central I/O planning will allow appropriate predictions about transfer times to and from the ad-hoc overlay file systems. As long as an application reads and writes only within its private ad-hoc overlay file system, it will use dedicated I/O resources in an exclusive manner. In addition, information about an application I/O behavior will be extracted. Such information is useful for planning the staging-in and staging-out of data as well as to optimize the data distribution within the ad-hoc overlay file system. This information will be provided by integrated monitoring components at the application level. The performance advantage by the proposed solution increases with the parallelism, provided that the combined bandwidths of local storage caches will always surpass the total bandwidth of central storage systems. The solutions proposed by ADA-FS will incorporate novel node-local storage locations forecasted for upcoming HPC installations.

提出的ADA-FS项目旨在提高极端规模并行应用的I/O性能。它将在系统软件中引入三个新颖的组件。首先，它提出了使用临时覆盖文件系统的概念。其次，设想一个中央I/O规划器组件。第三个组件集成了应用程序监视的概念，重点是应用程序的资源使用和I/O行为。临时覆盖文件系统将作为每个并行应用程序的单独实例部署在运行该应用程序的相同计算节点上。该文件系统将使用传统的和新颖的系统组件（如RAM、本地ssd或NVRAM）作为I/O子系统的数据缓存，前提是这些组件不被特定应用程序使用。这种方法将增加有效的I/O带宽并减少I/O操作的延迟。并行应用程序不需要修改就可以使用这个特别的文件系统。中央I/O规划器将协调所有与覆盖文件系统的所有实例之间的数据传输。规划器将防止I/O请求的随机混合，在这种情况下，应用程序无法预测全局共享I/O基础结构中任意给定时间点的总I/O带宽有多少可用。相反，中央I/O规划将允许对进出临时覆盖文件系统的传输时间进行适当的预测。只要应用程序仅在其专用的ad-hoc覆盖文件系统内读写，它就会以排他性的方式使用专用的I/O资源。此外，还将提取有关应用程序I/O行为的信息。这些信息对于规划数据的分入和分出以及优化临时覆盖文件系统中的数据分布非常有用。这些信息将由应用程序级别的集成监视组件提供。该方案的性能优势随着并行度的增加而增加，前提是本地存储缓存的组合带宽总是超过中央存储系统的总带宽。ADA-FS提出的解决方案将结合预测即将到来的HPC安装的新颖节点本地存储位置。