Efficient log data compression and analytics system

高效的日志数据压缩和分析系统

• 批准号: 570524-2021
• 负责人: Yuan, DingD
• 金额: $ 17.46万
• 依托单位: University of Toronto
• 依托单位国家: 加拿大
• 项目类别: Alliance Grants
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=751741
• 关键词: Efficient; log; data; compression; analytics

The pandemic has brought a furious growth of computer log data, i.e., the runtime data generated by computer systems. This is catalyzed by two trends. First, during the pandemic many activities are moved to online, hence there is an unprecedented growth of software services, which, in turn, generate an unprecedented amount of log data. In addition, software companies are increasingly relying on analysis of data to make data-driven decisions. They build large scale big data analytics systems that run hundreds of jobs per day. These data analytics systems end up generating even more log data. As a result, it is common for a company to generate Petabytes of log data per day, with 50% - 100% year-over-year growth.This unprecedented amount of log data creates a number of new challenges. First, conventional log analytics systems no longer scale to such large data sizes. In addition, managing such large data is extremely costly at every level, from storage cost, to network bandwidth, and to the cost of compute resources. This research proposes a novel system called CLP (Compressed Log Processor). CLP compresses the logs to unprecedented compression ratio, and more importantly, it allows one to search the compressed logs without decompression. The success of CLP will reduce the cost of log management and storage by over 40x, saving internet companies hundreds of millions of dollars per year. More importantly, it enables users to quickly search through Petabytes of logs efficiently, something that was not possible. The partnership with Uber will make CLP production-ready, and establish Canada as a leader in this emerging area of scalable and efficient log management. It will also result in increased employments of skilled software engineers in Canada, significant enhancements of the competitiveness of Canada's data center technology, and reduced emission of Canadian data centers.

大流行带来了计算机日志数据的迅猛增长，即，由计算机系统生成的运行时数据。这是由两种趋势催化的。首先，在疫情期间，许多活动都转移到网上，因此软件服务出现了前所未有的增长，这反过来又产生了前所未有的大量日志数据。此外，软件公司越来越依赖数据分析来做出数据驱动的决策。他们建立了大规模的大数据分析系统，每天运行数百个作业。这些数据分析系统最终会生成更多的日志数据。因此，公司每天生成PB级的日志数据是很常见的，并且每年增长50% - 100%。这种前所未有的日志数据量带来了许多新的挑战。首先，传统的日志分析系统不再扩展到如此大的数据大小。此外，管理如此庞大的数据在各个层面上都非常昂贵，从存储成本到网络带宽，再到计算资源成本。本研究提出一种新的系统称为CLP（压缩日志处理器）。CLP将日志压缩到前所未有的压缩比，更重要的是，它允许人们在不解压缩的情况下搜索压缩日志。CLP的成功将使日志管理和存储成本降低40倍以上，每年为互联网公司节省数亿美元。更重要的是，它使用户能够快速有效地搜索PB级的日志，这是不可能的。与Uber的合作将使CLP为生产做好准备，并使加拿大成为可扩展和高效日志管理这一新兴领域的领导者。它还将增加加拿大熟练软件工程师的就业机会，显著增强加拿大数据中心技术的竞争力，并减少加拿大数据中心的排放。