Frameworks: Collaborative Research: ChronoLog: A High-Performance Storage Infrastructure for Activity and Log Workloads

现代计算应用程序以前所未有的速度生成海量数据。除了简单地存储数据之外，一个越来越普遍的需求是存储活动数据，也称为日志数据，它描述的是发生的事情，而不是正在发生的事情。活动数据由计算系统、科学仪器、电子设备等生成，也由人类生成。活动数据的快速增长给当前的数据管理系统带来了超出其能力的压力，并成为高性能计算系统的致命性能瓶颈。该项目开发了ChronoLog，这是一个有效和高效地组织和存储活动数据的新型系统。ChronoLog利用现代存储硬件，并提供以用户为中心的插件和易于使用的界面，以提高工作效率。它将以各种方式惠及不同的社区，例如在金融交易中实现更好的欺诈检测，更快和更准确的天气预报和模拟，缩短医疗和生物工程数据的洞察时间，自主计算(例如驾驶)，以及更安全的网络和移动服务。ChronoLog使用物理时间来提供无同步的数据分布和日志的总排序。它首先利用多个存储层，例如存储级内存(例如3D XPoint)和新的闪存(例如NVMe固态硬盘)，通过日志自动分层透明地扩展日志。然后采用可调的并行访问模型，提供多写多读(MWMR)语义和高并发I/O，以充分利用多层存储环境。ChronoLog的创新设计通过尾部和历史操作之间的I/O隔离支持高性能数据访问，通过新开发的弹性存储功能支持高效的资源利用，并通过新的3D日志分布支持可扩展性。它通过充当强一致性的权威来源，并借助快速的追加和提交语义来促进数据处理流水线。它可以用作仲裁器，提供大量功能，如事务隔离和原子性、一致性复制和索引服务的共识引擎，以及可扩展的数据集成和仓库解决方案。ChronoLog及其插件建立了一个强大、灵活和高性能的存储生态系统，可促进高性能计算系统的可扩展应用程序和服务的开发。该项目包括一组不同的合作者，他们有着共同的需求，需要一种全新的分布式日志记录方法来解决他们的用例。这些密切的伙伴关系将加强学术和应用科学之间的联系，最终导致新的应用，并推动地球科学、宇宙学和天体物理学等不同领域的发现。形成这些合作并整合学生和初级IT专业人员将在网络基础设施方面创建一支训练有素的劳动力队伍。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Modern computing applications generate massive amounts of data at unprecedented rates. Beyond simply storing data, one increasingly common requirement is to store activity data, also known as log data, which describe things that happen rather than things that are. Activity data are generated by computing systems, scientific instruments, electrical devices, etc. as well as by humans. The fast growing of activity data stresses current data management systems beyond their capability and becomes a known killer performance bottleneck of high-performance computing systems. This project develops ChronoLog, a novel system for organizing and storing activity data effectively and efficiently. ChronoLog leverages modern storage hardware and provides user-focused plugins and easy-to-use interface for productivity. It will benefit a diverse range of communities in various ways, such as enabling better fraud detection in financial transactions, faster and more accurate weather predictions and simulations, reduced time-to-insight for medical and bioengineering data, autonomous computing (e.g., driving), and more secure web and mobile services. ChronoLog uses physical time to provide a synchronization-free data distribution and the total ordering on a log. It first leverages multiple storage tiers, such as storage-class memories (e.g., 3D XPoint) and new flash storage (e.g., NVMe SSDs), to transparently scale the log via log auto-tiering. It then adopts a tunable parallel access model, which offers multiple-writers-multiple-readers (MWMR) semantics and highly concurrent I/O, to fully utilize the multi-tiered storage environment. ChronoLog's innovative design supports high-performance data access via I/O isolation between tails and historical operations, efficient resource utilization with newly developed elastic storage capabilities, and scalability using a novel 3D log distribution. It facilitates data processing pipelining by acting as an authoritative source of strong consistency and with the help of fast append and commit semantics. It can be used as an arbitrator offering a plethora of features such as transactional isolation and atomicity, a consensus engine for consistent replication and indexing services, and a scalable data integration and warehousing solution. ChronoLog and its plugins establish a robust, flexible, and high-performance storage ecosystem that promotes the development of scalable applications and services for high performance computing systems. The project includes a diverse group of collaborators who share a common need for a fundamentally new approach to distributed logging to address their use cases. These close partnerships will strengthen the bonds between academic and applied science, ultimately leading to new applications and driving discovery in domains as diverse as geoscience, cosmology, and astrophysics. Forming these collaborations and integrating students and junior IT professionals will create a well-trained workforce in cyberinfrastructure.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.