A Scalable Middleware for Coordinating Data Streams on Clouds

本研究探讨了许多领域应用中数据流的可扩展协调问题，例如智能电网以支持不同的信息源。在这样一个系统中，除了监测基础设施以产生大量数据的传感器和仪器外，还涉及电气设备、组织、市场法规和天气的信息，以获得情况意识。协调涉及数据处理操作内部和操作之间的多源连续数据流的同步、传播和聚合。协调过程被表述为在云编程模型中运行操作符的数据流图。**为了实现可扩展的协调，该建议提倡两个关键原则：首先，应该在协调中间件中引入增量数据流处理，以处理频繁和小的更新；其次，由于数据的频率、格式、质量、因果关系和同步标准多种多样，在数据处理操作中经常出现偏差。当出现倾斜时，数据流的一部分需要比其他部分花费更长的时间来处理任务，从而减慢整个应用程序的速度。因此，检测偏差和调整协调中间件在实现域应用程序的整体可伸缩性方面起着另一个关键作用。**长期目标是研究一个协调平台来定义、管理和解决多源数据流的协调，并通过增强的中间件体系结构和软件组件为涉及广泛分布式数据服务的行业标准做出贡献。短期目标是精心设计一个可伸缩的协调中间件，将增量数据处理、监视和倾斜检测纳入整个控制平面，以提高响应能力和可伸缩性。本研究主要集中在四个方面：(1)嵌入在协调中间件中的增量数据处理。动态负载分配算法在增量数据上触发算子，平衡数据流中的计算和通信成本；(2)偏斜检测。这需要监控方法和统计模型来确定运行时的倾斜；(3)增强控制平面。控制平面利用监测和倾斜检测组件，并在发生倾斜时触发坐标过程的反应；(4)中间件体系结构促进了监控、检测和自适应协调循环中的通信。在系统架构设计、数据特性、通信基础设施和云环境给定的不确定性下，对该体系结构进行了评估。**本研究适用于智能电网、网络物理系统、远程通信和移动服务、卫生保健监测和社交网络等多种实际应用。研究结果将通过会议和期刊论文传播。该软件还将向公众开放源代码，以帮助特定应用程序的开发。更重要的是，本提案的获奖将有助于为研究生提供HQP培训，以完成6篇论文（2篇博士论文和4篇硕士论文）。此外，该合同还将为爱立信在蒙特利尔的长期合作奠定坚实的基础，为电信领域的高可用云服务开发通用监测和诊断框架。

This research explores the problem of scalable coordination of data streams in many domain applications, such as Smart Grids to support diverse information sources. In such a system, besides sensors and instruments monitoring the infrastructure to produce an avalanche of data, information on electrical equipment, organizations, market regulations and weather are also engaged to gain situation awareness. Coordination involves synchronization, dissemination, and aggregation of multi-source, continuous data streams in and between data processing operations. A coordination process is formulated as a graph of data flows that running operators in cloud programming models. **To achieve scalable coordination, this proposal advocates two key principles: first, incremental data stream processing should be introduced to the coordination middleware to handle frequent and small updates; secondly, since data frequency, format, quality, causality and synchronization criteria are diverse, skew often occurs in data processing operations. When skew arises, a portion of a data flow takes significantly longer time to process the task than others and thus slows down the entire application. Therefore, detecting skews and adapting the coordination middleware play another key role in achieving overall scalability of a domain application. **The long term objective is to investigate a coordination platform to define, manage and resolve coordination of multi-source data streams, and contribute to industry standards involving broad distributed data services with enhanced middleware architecture and software components. The short term goal is to elaborate a scalable coordination middleware that engages incremental data processing, monitoring and skew detection into a whole control plane to improve responsiveness and scalability. This research program focuses on four integral aspects: (1) incremental data processing that is embedded in the coordination middleware. A dynamic load distribution algorithm triggers operators on incremental data and balances the computation and communication costs in the data flow; (2) skew detection. This requires monitoring methods and statistical models to determine a skew at runtime; (3) enhanced control plane. The control plane utilizes the monitoring and skew detection components and triggers reaction of the coordinate process when a skew occurs; and (4) middleware architecture facilitates communications in the coordination loop of monitoring, detecting and adapting. The architecture is evaluated under given uncertainties in the system architecture design, data characters, communication infrastructure and cloud environment. **The research is applicable to a variety of practical applications in smart grids, cyber-physical systems, tele-communication and mobile services, health care monitoring and social networking. The research results will be disseminated through conferences and journal papers. The software will also be made open source to the public to help application specific development. More important, the award of this proposal will help provide HQP training for graduate students to complete six theses (two PhD and four MASc). In addition, the award will also help establish a strong position to a long-term collaboration with Ericsson in Montreal on developing a general monitoring and diagnosis framework for high available cloud services in the telecommunication domain.