Unified Stream and Transaction Processing

点击翻译按钮获取中文摘要

The diversity and complexity data-processing architectures, as used in current large scale e-commerce and social media applications, hinder the integration of new data analytics features and increase maintenance and operational costs. These applications typically span three different layers, each one adopting a different processing model. Online transaction processing (OLTP) systems are first used to access core application data in a synchronous fashion. In parallel, events of significance are propagated to back-ends for data analysis. There, events are stored to persistent storage for offline batch processing, e.g., using Hadoop clusters. Increasingly application providers are also deploying stream processing solutions to process data in near real-time and improve the freshness of data analysis. Results from batch or real-time analysis are then propagated back to serving layers in online front-end systems for applications that require them.The fragmentation of application logic across the transaction, stream and batch processing models increases operational costs and application development time. Substantial engineering effort is required to export, move, convert, and import data across different systems when performing exploratory analytics for developing new applications. Developers do not have a reliable way of mapping application features on a specific processing model, re- lying on past experience and rule of thumb. Reimplementing complex processing pipelines in a system that adopts a different way of thinking is costly and the performance trade-offs among them are difficult to estimate.The operation of multiple systems in parallel leads to increased operational costs. Operating different platforms either requires allocating computing resources statically to the different systems, which is not flexible, or to use cluster management platformssuch as Mesos or YARN. However to accommodate diverse systems these platforms necessarily provide a restricted management interface which limits their ability to operate on the basis of detailed application metrics.The goal of this proposal is to unify two of these models, stream and transaction process- ing by i) exploring performance trade-offs between the two models, ii) designing a processing model that sup- ports both, iii) implementing a system prototype which could provide good performance on both transactional and stream processing workloads, and iv) prototyping mixed processing applications which requires both stream and transaction processing functionalities.