Unified Stream and Transaction Processing

当前大规模电子商务和社交媒体应用中使用的数据处理架构的多样性和复杂性阻碍了新数据分析功能的集成，并增加了维护和运营成本。这些应用程序通常跨越三个不同的层，每个层采用不同的处理模型。联机事务处理（OLTP）系统首先用于以同步方式访问核心应用程序数据。同时，重要事件被传播到后端进行数据分析。在那里，事件被存储到持久化存储以进行离线批处理，例如，使用Hadoop集群。越来越多的应用程序提供商也在部署流处理解决方案来近乎实时地处理数据，并提高数据分析的新鲜度。然后将批处理或实时分析的结果传播回在线前端系统中的服务层，以供需要它们的应用程序使用。跨事务、流和批处理模型的应用程序逻辑碎片增加了操作成本和应用程序开发时间。在为开发新应用程序执行探索性分析时，需要大量的工程工作来跨不同系统导出、移动、转换和导入数据。开发人员依靠过去的经验和经验法则，没有可靠的方法将应用程序的特性映射到特定的处理模型上。在采用不同思维方式的系统中重新实现复杂的处理管道是昂贵的，并且它们之间的性能权衡很难估计。多个系统并行运行导致运营成本增加。操作不同的平台要么需要将计算资源静态地分配给不同的系统，这是不灵活的，要么需要使用集群管理平台，如Mesos或YARN。然而，为了适应不同的系统，这些平台必须提供一个受限制的管理接口，这限制了它们基于详细的应用程序度量进行操作的能力。本提案的目标是统一这两种模型，流处理和事务处理——通过i)探索两种模型之间的性能权衡，ii)设计一个支持这两种模型的处理模型，iii)实现一个可以在事务处理和流处理工作负载上提供良好性能的系统原型，iv)原型化混合处理应用程序，它需要流和事务处理功能。

The diversity and complexity data-processing architectures, as used in current large scale e-commerce and social media applications, hinder the integration of new data analytics features and increase maintenance and operational costs. These applications typically span three different layers, each one adopting a different processing model. Online transaction processing (OLTP) systems are first used to access core application data in a synchronous fashion. In parallel, events of significance are propagated to back-ends for data analysis. There, events are stored to persistent storage for offline batch processing, e.g., using Hadoop clusters. Increasingly application providers are also deploying stream processing solutions to process data in near real-time and improve the freshness of data analysis. Results from batch or real-time analysis are then propagated back to serving layers in online front-end systems for applications that require them.The fragmentation of application logic across the transaction, stream and batch processing models increases operational costs and application development time. Substantial engineering effort is required to export, move, convert, and import data across different systems when performing exploratory analytics for developing new applications. Developers do not have a reliable way of mapping application features on a specific processing model, re- lying on past experience and rule of thumb. Reimplementing complex processing pipelines in a system that adopts a different way of thinking is costly and the performance trade-offs among them are difficult to estimate.The operation of multiple systems in parallel leads to increased operational costs. Operating different platforms either requires allocating computing resources statically to the different systems, which is not flexible, or to use cluster management platformssuch as Mesos or YARN. However to accommodate diverse systems these platforms necessarily provide a restricted management interface which limits their ability to operate on the basis of detailed application metrics.The goal of this proposal is to unify two of these models, stream and transaction process- ing by i) exploring performance trade-offs between the two models, ii) designing a processing model that sup- ports both, iii) implementing a system prototype which could provide good performance on both transactional and stream processing workloads, and iv) prototyping mixed processing applications which requires both stream and transaction processing functionalities.