A Scalable Middleware for Coordinating Data Streams on Clouds

本研究探讨了许多领域应用中数据流的可伸缩协调问题，例如支持不同信息源的智能电网。在这样的系统中，除了监测基础设施以产生雪崩数据的传感器和仪器外，还利用有关电气设备、组织、市场法规和天气的信息来获得情况感知。协调涉及数据处理操作内部和之间的多源、连续数据流的同步、传播和聚合。协调过程被描述为在云编程模型中运行运算符的数据流的图。为了实现可扩展的协调，该方案倡导两个关键原则：一是在协调中间件中引入增量数据流处理，以处理频繁和少量的更新；二是由于数据的频率、格式、质量、因果关系和同步标准各不相同，数据处理操作中经常会出现偏差。当出现不对称时，数据流的一部分处理任务所需的时间比其他部分长得多，从而降低了整个应用程序的运行速度。因此，检测偏差和调整协调中间件在实现域应用程序的整体可伸缩性方面扮演着另一个关键角色。长期目标是研究一个协调平台，以定义、管理和解决多源数据流的协调，并通过增强的中间件体系结构和软件组件促进涉及广泛分布式数据服务的行业标准。短期目标是精心设计一种可扩展的协调中间件，将增量数据处理、监控和偏差检测整合到整个控制平面中，以提高响应性和可扩展性。本研究方案主要集中在四个方面：(1)嵌入到协同中间件中的增量数据处理。动态负载分配算法在增量数据上触发算子，平衡数据流中的计算和通信代价；(2)偏差检测。这需要监控方法和统计模型来确定运行时的偏差；(3)增强的控制平面。控制平面利用监控和偏斜检测组件，在偏斜发生时触发协调过程的反应；(4)中间件架构便于监控、检测和适配的协调循环中的通信。在给定系统架构设计、数据特征、通信基础设施和云环境等不确定性的情况下，对该架构进行了评估。该研究适用于智能电网、网络物理系统、电信和移动服务、医疗保健监测和社交网络等各种实际应用。研究成果将通过会议和期刊论文传播。该软件还将向公众开放源代码，以帮助特定应用程序的开发。更重要的是，这项提议的授予将有助于为研究生提供完成六篇论文(两名博士和四名硕士)的HQP培训。此外，该奖项还将有助于与蒙特利尔爱立信建立长期合作的有利地位，为电信领域的高可用性云服务开发通用监测和诊断框架。

This research explores the problem of scalable coordination of data streams in many domain applications, such as Smart Grids to support diverse information sources. In such a system, besides sensors and instruments monitoring the infrastructure to produce an avalanche of data, information on electrical equipment, organizations, market regulations and weather are also engaged to gain situation awareness. Coordination involves synchronization, dissemination, and aggregation of multi-source, continuous data streams in and between data processing operations. A coordination process is formulated as a graph of data flows that running operators in cloud programming models. To achieve scalable coordination, this proposal advocates two key principles: first, incremental data stream processing should be introduced to the coordination middleware to handle frequent and small updates; secondly, since data frequency, format, quality, causality and synchronization criteria are diverse, skew often occurs in data processing operations. When skew arises, a portion of a data flow takes significantly longer time to process the task than others and thus slows down the entire application. Therefore, detecting skews and adapting the coordination middleware play another key role in achieving overall scalability of a domain application. The long term objective is to investigate a coordination platform to define, manage and resolve coordination of multi-source data streams, and contribute to industry standards involving broad distributed data services with enhanced middleware architecture and software components. The short term goal is to elaborate a scalable coordination middleware that engages incremental data processing, monitoring and skew detection into a whole control plane to improve responsiveness and scalability. This research program focuses on four integral aspects: (1) incremental data processing that is embedded in the coordination middleware. A dynamic load distribution algorithm triggers operators on incremental data and balances the computation and communication costs in the data flow; (2) skew detection. This requires monitoring methods and statistical models to determine a skew at runtime; (3) enhanced control plane. The control plane utilizes the monitoring and skew detection components and triggers reaction of the coordinate process when a skew occurs; and (4) middleware architecture facilitates communications in the coordination loop of monitoring, detecting and adapting. The architecture is evaluated under given uncertainties in the system architecture design, data characters, communication infrastructure and cloud environment. The research is applicable to a variety of practical applications in smart grids, cyber-physical systems, tele-communication and mobile services, health care monitoring and social networking. The research results will be disseminated through conferences and journal papers. The software will also be made open source to the public to help application specific development. More important, the award of this proposal will help provide HQP training for graduate students to complete six theses (two PhD and four MASc). In addition, the award will also help establish a strong position to a long-term collaboration with Ericsson in Montreal on developing a general monitoring and diagnosis framework for high available cloud services in the telecommunication domain.