A Scalable Middleware for Coordinating Data Streams on Clouds

点击翻译按钮获取中文摘要

This research explores the problem of scalable coordination of data streams in many domain applications, such as Smart Grids to support diverse information sources. In such a system, besides sensors and instruments monitoring the infrastructure to produce an avalanche of data, information on electrical equipment, organizations, market regulations and weather are also engaged to gain situation awareness. Coordination involves synchronization, dissemination, and aggregation of multi-source, continuous data streams in and between data processing operations. A coordination process is formulated as a graph of data flows that running operators in cloud programming models. To achieve scalable coordination, this proposal advocates two key principles: first, incremental data stream processing should be introduced to the coordination middleware to handle frequent and small updates; secondly, since data frequency, format, quality, causality and synchronization criteria are diverse, skew often occurs in data processing operations. When skew arises, a portion of a data flow takes significantly longer time to process the task than others and thus slows down the entire application. Therefore, detecting skews and adapting the coordination middleware play another key role in achieving overall scalability of a domain application. The long term objective is to investigate a coordination platform to define, manage and resolve coordination of multi-source data streams, and contribute to industry standards involving broad distributed data services with enhanced middleware architecture and software components. The short term goal is to elaborate a scalable coordination middleware that engages incremental data processing, monitoring and skew detection into a whole control plane to improve responsiveness and scalability. This research program focuses on four integral aspects: (1) incremental data processing that is embedded in the coordination middleware. A dynamic load distribution algorithm triggers operators on incremental data and balances the computation and communication costs in the data flow; (2) skew detection. This requires monitoring methods and statistical models to determine a skew at runtime; (3) enhanced control plane. The control plane utilizes the monitoring and skew detection components and triggers reaction of the coordinate process when a skew occurs; and (4) middleware architecture facilitates communications in the coordination loop of monitoring, detecting and adapting. The architecture is evaluated under given uncertainties in the system architecture design, data characters, communication infrastructure and cloud environment. The research is applicable to a variety of practical applications in smart grids, cyber-physical systems, tele-communication and mobile services, health care monitoring and social networking. The research results will be disseminated through conferences and journal papers. The software will also be made open source to the public to help application specific development. More important, the award of this proposal will help provide HQP training for graduate students to complete six theses (two PhD and four MASc). In addition, the award will also help establish a strong position to a long-term collaboration with Ericsson in Montreal on developing a general monitoring and diagnosis framework for high available cloud services in the telecommunication domain.