权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

Unified Stream and Transaction Processing

统一流和事务处理

基本信息

批准号：
EP/N000110/1
负责人：
Matteo Migliavacca
金额：
$ 12.16万
依托单位：
University of Kent
依托单位国家：
英国
项目类别：
Research Grant
财政年份：
2016
资助国家：
英国
起止时间：
2016 至无数据
项目状态：
已结题

来源：
https://gtr.ukri.org/projects?ref=EP%2FN000110%2F1
关键词：
Unified Stream Transaction Processing

项目摘要

The diversity and complexity data-processing architectures, as used in current large scale e-commerce and social media applications, hinder the integration of new data analytics features and increase maintenance and operational costs. These applications typically span three different layers, each one adopting a different processing model. Online transaction processing (OLTP) systems are first used to access core application data in a synchronous fashion. In parallel, events of significance are propagated to back-ends for data analysis. There, events are stored to persistent storage for offline batch processing, e.g., using Hadoop clusters. Increasingly application providers are also deploying stream processing solutions to process data in near real-time and improve the freshness of data analysis. Results from batch or real-time analysis are then propagated back to serving layers in online front-end systems for applications that require them.The fragmentation of application logic across the transaction, stream and batch processing models increases operational costs and application development time. Substantial engineering effort is required to export, move, convert, and import data across different systems when performing exploratory analytics for developing new applications. Developers do not have a reliable way of mapping application features on a specific processing model, re- lying on past experience and rule of thumb. Reimplementing complex processing pipelines in a system that adopts a different way of thinking is costly and the performance trade-offs among them are difficult to estimate.The operation of multiple systems in parallel leads to increased operational costs. Operating different platforms either requires allocating computing resources statically to the different systems, which is not flexible, or to use cluster management platformssuch as Mesos or YARN. However to accommodate diverse systems these platforms necessarily provide a restricted management interface which limits their ability to operate on the basis of detailed application metrics.The goal of this proposal is to unify two of these models, stream and transaction process- ing by i) exploring performance trade-offs between the two models, ii) designing a processing model that sup- ports both, iii) implementing a system prototype which could provide good performance on both transactional and stream processing workloads, and iv) prototyping mixed processing applications which requires both stream and transaction processing functionalities.

当前大规模电子商务和社交媒体应用程序中使用的数据处理架构的多样性和复杂性阻碍了新数据分析功能的整合，并增加了维护和运营成本。这些应用程序通常跨越三个不同的层，每个层采用不同的处理模型。在线事务处理(OLTP)系统首先用于以同步方式访问核心应用程序数据。同时，重要事件被传播到后端以进行数据分析。在那里，事件被存储到永久存储器以用于离线批处理，例如使用Hadoop集群。越来越多的应用程序提供商也在部署流处理解决方案，以接近实时地处理数据并提高数据分析的新鲜度。然后，批处理或实时分析的结果被传播回在线前端系统中的服务层，以供需要它们的应用程序使用。跨事务、流和批处理模型的应用程序逻辑碎片化增加了运营成本和应用程序开发时间。在执行探索性分析以开发新应用程序时，跨不同系统导出、移动、转换和导入数据需要大量的工程工作。开发人员没有一种可靠的方法将应用程序功能映射到特定的处理模型上，依赖于过去的经验和经验法则。在采用不同思维方式的系统中重新实现复杂的处理流水线成本很高，并且它们之间的性能权衡很难估计。多个系统的并行运行会导致运营成本增加。运行不同的平台需要将计算资源静态地分配给不同的系统，这是不灵活的，或者使用集群管理平台，如Mesos或YAR。然而，为了适应不同的系统，这些平台必须提供受限的管理接口，这限制了它们基于详细的应用度量进行操作的能力。本方案的目标是通过i)探索两种模型之间的性能权衡，ii)设计支持两者的处理模型，iii)实现能够在事务和流处理工作负载上都提供良好性能的系统原型，以及iv)原型化需要流和事务处理功能的混合处理应用来统一这两种模型。