ADA-FS - Advanced Data Placement via Ad-hoc File Systems at Extreme Scales

建议的ADA-FS项目旨在提高极端规模并行应用的I/O性能。它将在系统软件中引入三个新的组件。首先，它提出了使用特别覆盖文件系统的概念。其次，设想了一个中央I/O规划器组件。第三个组件集成了应用程序监视的概念，重点放在其资源使用和I/O行为上。临时覆盖文件系统将作为每个并行应用程序的单独实例部署在运行该应用程序的相同计算节点上。此文件系统将使用传统的和新颖的系统组件，如RAM、本地SSD或NVRAM作为I/O子系统的数据缓存，前提是这些组件未被特定应用程序使用。这种方法将增加有效的I/O带宽，并减少I/O操作的延迟。并行应用程序不需要修改即可使用此临时文件系统。中央I/O规划器将协调与覆盖文件系统的所有实例之间的所有数据传输。规划器将防止随机混合的I/O请求，在这种情况下，任何应用程序都无法预测在全局共享I/O基础架构中的任何给定时间点有多少总I/O带宽可用。相反，中央I/O规划将允许对进出临时覆盖文件系统的传输时间进行适当的预测。只要应用程序仅在其专用临时覆盖文件系统内进行读写，它就会以独占方式使用专用I/O资源。此外，还将提取有关应用程序I/O行为的信息。这样的信息对于规划数据的暂入和暂出以及优化自组织覆盖文件系统内的数据分布是有用的。这些信息将由应用程序一级的综合监测组件提供。如果本地存储缓存的总带宽始终超过中央存储系统的总带宽，则建议的解决方案的性能优势随着并行度的增加而增加。ADA-FS提出的解决方案将结合为即将到来的HPC安装预测的新的节点本地存储位置。

The proposed ADA-FS project aims at improving the I/O performance for extreme-scale parallel applications. It will introduce three novel components to the system software. First, it proposes the use of an ad-hoc overlay file system concept. Second, a central I/O planner component is envisioned. The third component integrates a concept for application monitoring, with emphasis on its resource usage and I/O behavior. The ad-hoc overlay file system will be deployed as a separate instance for every parallel application on the same compute nodes that the application is running on. This file system will use traditional and novel system components such as RAM, local SSDs, or NVRAM as data caches for the I/O subsystem provided that these are not in use by the particular application. This approach will increase the effective I/O bandwidth and reduce the latency of I/O operations. Parallel applications will require no modifications to make use of this ad-hoc file system. The central I/O planner will coordinate all data transfers to and from all instances of the overlay file system. The planner will prevent a random mix of I/O requests in which no application can anticipate how much of the total I/O bandwidth is available at any given point in time in the globally shared I/O infrastructure. Instead, the central I/O planning will allow appropriate predictions about transfer times to and from the ad-hoc overlay file systems. As long as an application reads and writes only within its private ad-hoc overlay file system, it will use dedicated I/O resources in an exclusive manner. In addition, information about an application I/O behavior will be extracted. Such information is useful for planning the staging-in and staging-out of data as well as to optimize the data distribution within the ad-hoc overlay file system. This information will be provided by integrated monitoring components at the application level. The performance advantage by the proposed solution increases with the parallelism, provided that the combined bandwidths of local storage caches will always surpass the total bandwidth of central storage systems. The solutions proposed by ADA-FS will incorporate novel node-local storage locations forecasted for upcoming HPC installations.