Efficient log data compression and analytics system

点击翻译按钮获取中文摘要

The pandemic has brought a furious growth of computer log data, i.e., the runtime data generated by computer systems. This is catalyzed by two trends. First, during the pandemic many activities are moved to online, hence there is an unprecedented growth of software services, which, in turn, generate an unprecedented amount of log data. In addition, software companies are increasingly relying on analysis of data to make data-driven decisions. They build large scale big data analytics systems that run hundreds of jobs per day. These data analytics systems end up generating even more log data. As a result, it is common for a company to generate Petabytes of log data per day, with 50% - 100% year-over-year growth.This unprecedented amount of log data creates a number of new challenges. First, conventional log analytics systems no longer scale to such large data sizes. In addition, managing such large data is extremely costly at every level, from storage cost, to network bandwidth, and to the cost of compute resources. This research proposes a novel system called CLP (Compressed Log Processor). CLP compresses the logs to unprecedented compression ratio, and more importantly, it allows one to search the compressed logs without decompression. The success of CLP will reduce the cost of log management and storage by over 40x, saving internet companies hundreds of millions of dollars per year. More importantly, it enables users to quickly search through Petabytes of logs efficiently, something that was not possible. The partnership with Uber will make CLP production-ready, and establish Canada as a leader in this emerging area of scalable and efficient log management. It will also result in increased employments of skilled software engineers in Canada, significant enhancements of the competitiveness of Canada's data center technology, and reduced emission of Canadian data centers.