Frameworks: Collaborative Research: ChronoLog: A High-Performance Storage Infrastructure for Activity and Log Workloads

点击翻译按钮获取中文摘要

Modern computing applications generate massive amounts of data at unprecedented rates. Beyond simply storing data, one increasingly common requirement is to store activity data, also known as log data, which describe things that happen rather than things that are. Activity data are generated by computing systems, scientific instruments, electrical devices, etc. as well as by humans. The fast growing of activity data stresses current data management systems beyond their capability and becomes a known killer performance bottleneck of high-performance computing systems. This project develops ChronoLog, a novel system for organizing and storing activity data effectively and efficiently. ChronoLog leverages modern storage hardware and provides user-focused plugins and easy-to-use interface for productivity. It will benefit a diverse range of communities in various ways, such as enabling better fraud detection in financial transactions, faster and more accurate weather predictions and simulations, reduced time-to-insight for medical and bioengineering data, autonomous computing (e.g., driving), and more secure web and mobile services. ChronoLog uses physical time to provide a synchronization-free data distribution and the total ordering on a log. It first leverages multiple storage tiers, such as storage-class memories (e.g., 3D XPoint) and new flash storage (e.g., NVMe SSDs), to transparently scale the log via log auto-tiering. It then adopts a tunable parallel access model, which offers multiple-writers-multiple-readers (MWMR) semantics and highly concurrent I/O, to fully utilize the multi-tiered storage environment. ChronoLog's innovative design supports high-performance data access via I/O isolation between tails and historical operations, efficient resource utilization with newly developed elastic storage capabilities, and scalability using a novel 3D log distribution. It facilitates data processing pipelining by acting as an authoritative source of strong consistency and with the help of fast append and commit semantics. It can be used as an arbitrator offering a plethora of features such as transactional isolation and atomicity, a consensus engine for consistent replication and indexing services, and a scalable data integration and warehousing solution. ChronoLog and its plugins establish a robust, flexible, and high-performance storage ecosystem that promotes the development of scalable applications and services for high performance computing systems. The project includes a diverse group of collaborators who share a common need for a fundamentally new approach to distributed logging to address their use cases. These close partnerships will strengthen the bonds between academic and applied science, ultimately leading to new applications and driving discovery in domains as diverse as geoscience, cosmology, and astrophysics. Forming these collaborations and integrating students and junior IT professionals will create a well-trained workforce in cyberinfrastructure.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.