Frameworks: Collaborative Research: ChronoLog: A High-Performance Storage Infrastructure for Activity and Log Workloads

框架：协作研究：ChronoLog：用于活动和日志工作负载的高性能存储基础架构

• 批准号: 2104013
• 负责人: Xian-He Sun
• 金额: $ 267.65万
• 依托单位: Illinois Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-06-01 至 2025-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2104013&HistoricalAwards=false
• 关键词: Frameworks; Collaborative; Research; ChronoLog; Performance

Modern computing applications generate massive amounts of data at unprecedented rates. Beyond simply storing data, one increasingly common requirement is to store activity data, also known as log data, which describe things that happen rather than things that are. Activity data are generated by computing systems, scientific instruments, electrical devices, etc. as well as by humans. The fast growing of activity data stresses current data management systems beyond their capability and becomes a known killer performance bottleneck of high-performance computing systems. This project develops ChronoLog, a novel system for organizing and storing activity data effectively and efficiently. ChronoLog leverages modern storage hardware and provides user-focused plugins and easy-to-use interface for productivity. It will benefit a diverse range of communities in various ways, such as enabling better fraud detection in financial transactions, faster and more accurate weather predictions and simulations, reduced time-to-insight for medical and bioengineering data, autonomous computing (e.g., driving), and more secure web and mobile services. ChronoLog uses physical time to provide a synchronization-free data distribution and the total ordering on a log. It first leverages multiple storage tiers, such as storage-class memories (e.g., 3D XPoint) and new flash storage (e.g., NVMe SSDs), to transparently scale the log via log auto-tiering. It then adopts a tunable parallel access model, which offers multiple-writers-multiple-readers (MWMR) semantics and highly concurrent I/O, to fully utilize the multi-tiered storage environment. ChronoLog's innovative design supports high-performance data access via I/O isolation between tails and historical operations, efficient resource utilization with newly developed elastic storage capabilities, and scalability using a novel 3D log distribution. It facilitates data processing pipelining by acting as an authoritative source of strong consistency and with the help of fast append and commit semantics. It can be used as an arbitrator offering a plethora of features such as transactional isolation and atomicity, a consensus engine for consistent replication and indexing services, and a scalable data integration and warehousing solution. ChronoLog and its plugins establish a robust, flexible, and high-performance storage ecosystem that promotes the development of scalable applications and services for high performance computing systems. The project includes a diverse group of collaborators who share a common need for a fundamentally new approach to distributed logging to address their use cases. These close partnerships will strengthen the bonds between academic and applied science, ultimately leading to new applications and driving discovery in domains as diverse as geoscience, cosmology, and astrophysics. Forming these collaborations and integrating students and junior IT professionals will create a well-trained workforce in cyberinfrastructure.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代计算应用程序以前所未有的速率产生大量数据。除了简单地存储数据外，一个日益普遍的要求是存储活动数据（也称为日志数据），这些数据描述了发生的事情而不是事物。活动数据是通过计算系统，科学仪器，电气设备等以及人类生成的。活动数据的快速增长强调了当前的数据管理系统超出其能力，并成为高性能计算系统的已知杀手性能瓶颈。该项目开发时间表，这是一个有效，有效地组织和存储活动数据的新型系统。按时间顺序利用现代存储硬件，并提​​供以用户为中心的插件和易于使用的界面以提高生产力。它将以各种方式使各种各样的社区受益，例如在金融交易中更好地欺诈检测，更快，更准确的天气预测和模拟，减少医疗和生物工程数据的时间段落，自动驾驶计算（例如驾驶）以及更安全的网络和移动服务。按钮使用物理时间来提供无同步的数据分布和日志上的总订购。它首先利用多个存储层，例如存储级记忆（例如3D Xpoint）和新的闪存存储（例如NVME SSD），以通过日志自动层透明地扩展日志。然后，它采用可调的并行访问模型，该模型提供了多个编织者 - 读取器（MWMR）语义和高度并发I/O，以充分利用多层存储环境。按Chronolog的创新设计通过I/O之间的I/O隔离，具有新开发的弹性存储功能的有效资源利用以及使用新颖的3D日志分布的可扩展性来支持高性能数据访问。它通过充当强大一致性的权威来源并借助快速附加和提交语义来促进数据处理。它可以用作提供多种功能的仲裁员，例如交易隔离和原子能，共识引擎，用于一致的复制和索引服务，以及可扩展的数据集成和仓库解决方案。按钮及其插件建立了一个健壮，灵活和高性能的存储生态系统，可促进高性能计算系统的可扩展应用程序和服务的开发。该项目包括一组各种各样的合作者，他们共同需要一种从根本上进行分发日志记录以解决其用例的新方法。这些紧密的伙伴关系将加强学术和应用科学之间的纽带，最终导致新的应用并推动在地球科学，宇宙学和天体物理学等领域的发现。组成这些合作并整合学生和初级IT专业人员将创建一个训练有素的网络基础设施劳动力。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估评估标准来通过评估来获得支持的。

项目成果

An Evaluation of DAOS for Simulation and Deep Learning HPC Workloads

• DOI: 10.1145/3578353.3589542
• 发表时间: 2023-05
• 期刊: Proceedings of the 3rd Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems
• 作者: Luke Logan;J. Lofstead;Xian-He Sun;Anthony Kougkas