A Scalable Middleware for Coordinating Data Streams on Clouds

本研究探讨在许多领域的应用，如智能电网，以支持不同的信息源的数据流的可扩展协调的问题。在这样一个系统中，除了监测基础设施以产生大量数据的传感器和仪器外，还利用关于电气设备、组织、市场法规和天气的信息来获得情况意识。协调涉及数据处理操作中和之间的多源、连续数据流的同步、传播和聚合。协调过程被公式化为在云编程模型中运行操作符的数据流图。为了实现可扩展的协调，该建议倡导两个关键原则：第一，增量数据流处理应引入到协调中间件，以处理频繁和小的更新;第二，由于数据的频率，格式，质量，因果关系和同步标准是不同的，偏斜经常发生在数据处理操作。当偏斜出现时，数据流的一部分比其他部分花费更长的时间来处理任务，从而降低整个应用程序的速度。因此，检测偏差和调整协调中间件在实现域应用程序的整体可伸缩性方面发挥着另一个关键作用。长期目标是研究一个协调平台，以定义、管理和解决多源数据流的协调，并为涉及广泛的分布式数据服务的行业标准做出贡献，这些服务具有增强的中间件架构和软件组件。短期目标是精心设计一个可扩展的协调中间件，将增量数据处理，监控和偏斜检测纳入整个控制平面，以提高响应能力和可扩展性。本研究计划主要集中在四个方面：（1）嵌入在协调中间件中的增量数据处理。动态负载分配算法在增量数据上触发操作符，平衡数据流中的计算和通信开销;（2）偏斜检测。这需要监控方法和统计模型来确定运行时的偏斜;（3）增强的控制平面。控制平面利用监视和偏斜检测组件，并且当偏斜发生时触发协调过程的反应;以及（4）中间件架构促进监视、检测和适配的协调环路中的通信。在系统架构设计、数据特征、通信基础设施和云环境中给定不确定性的情况下，对该架构进行评估。该研究适用于智能电网，网络物理系统，远程通信和移动的服务，医疗保健监测和社交网络的各种实际应用。研究成果将通过会议和期刊论文传播。该软件还将向公众开放源代码，以帮助特定应用程序的开发。更重要的是，该提案的授予将有助于为研究生提供HQP培训，以完成六篇论文（两篇博士和四篇硕士）。此外，该奖项还将有助于与爱立信在蒙特利尔建立长期合作关系，为电信领域的高可用云服务开发通用监控和诊断框架。

This research explores the problem of scalable coordination of data streams in many domain applications, such as Smart Grids to support diverse information sources. In such a system, besides sensors and instruments monitoring the infrastructure to produce an avalanche of data, information on electrical equipment, organizations, market regulations and weather are also engaged to gain situation awareness. Coordination involves synchronization, dissemination, and aggregation of multi-source, continuous data streams in and between data processing operations. A coordination process is formulated as a graph of data flows that running operators in cloud programming models. To achieve scalable coordination, this proposal advocates two key principles: first, incremental data stream processing should be introduced to the coordination middleware to handle frequent and small updates; secondly, since data frequency, format, quality, causality and synchronization criteria are diverse, skew often occurs in data processing operations. When skew arises, a portion of a data flow takes significantly longer time to process the task than others and thus slows down the entire application. Therefore, detecting skews and adapting the coordination middleware play another key role in achieving overall scalability of a domain application. The long term objective is to investigate a coordination platform to define, manage and resolve coordination of multi-source data streams, and contribute to industry standards involving broad distributed data services with enhanced middleware architecture and software components. The short term goal is to elaborate a scalable coordination middleware that engages incremental data processing, monitoring and skew detection into a whole control plane to improve responsiveness and scalability. This research program focuses on four integral aspects: (1) incremental data processing that is embedded in the coordination middleware. A dynamic load distribution algorithm triggers operators on incremental data and balances the computation and communication costs in the data flow; (2) skew detection. This requires monitoring methods and statistical models to determine a skew at runtime; (3) enhanced control plane. The control plane utilizes the monitoring and skew detection components and triggers reaction of the coordinate process when a skew occurs; and (4) middleware architecture facilitates communications in the coordination loop of monitoring, detecting and adapting. The architecture is evaluated under given uncertainties in the system architecture design, data characters, communication infrastructure and cloud environment. The research is applicable to a variety of practical applications in smart grids, cyber-physical systems, tele-communication and mobile services, health care monitoring and social networking. The research results will be disseminated through conferences and journal papers. The software will also be made open source to the public to help application specific development. More important, the award of this proposal will help provide HQP training for graduate students to complete six theses (two PhD and four MASc). In addition, the award will also help establish a strong position to a long-term collaboration with Ericsson in Montreal on developing a general monitoring and diagnosis framework for high available cloud services in the telecommunication domain.