Parallel complex event processing to meet probabilistic latency bounds II

点击翻译按钮获取中文摘要

Meeting latency bounds in complex event processing (CEP) in a consistent and cost-efficient manner is highly challenging. Operators have to be capable of processing events at very high rate and also under varying workload. This requires methods to parallelize multiple dependent operators and dynamically adapt the configuration of these operators, e.g., by changing the parallelization degree and resources used in each operators' configuration. As a result, in the first phase of this project (Precept I) we developed methods for fine-grained parallelization of CEP operators that can probabilistically meet a latency bound under heavily fluctuating and dynamic workloads. In Precept I, we approached the problem under the assumption that there are unlimited resources available. However, there are situations where the monetary budget and/or the available resources for running a CEP system are limited and additional mechanisms must be employed to meet the lateny bounds of the system. In order to address the aforementioned problem, in this proposal (Precept II), we aim to shed load at the operators so that a given latency bound for a CEP system can be met despite resource constraints. While shedding load may have an adverse impact on the quality of complex event processing, we aim to develop load shedding strategies for operators that maximize the quality of result perceived by the application and also meet the required latency bounds. Our load shedding strategies should support a wide spectrum of CEP operators. Furthermore, we aim to combine the optimization scheme without load shedding (Precept I) and the scheme with load shedding (Precept II) such that, for any input condition, the system maximizes quality while maintaining the given latency bound, and minimizes cost while the resource limit is not reached. The proposed methods and concepts will be evaluated using real-world workloads of camera networks and traffic monitoring.