Approximate Distributed Stream Tracking: Enabling the Next Generation of Data-Streaming Applications

本研究开发了模型、算法方法和软件解决方案，用于跟踪用于监控应用的海量数据流，如IP（互联网协议）网络流量分析。这种监视应用程序本质上是分布式的，依赖于关联多个流，因此在严重的通信约束方面提出了挑战。此外，即使在集中式或单流情况下，这种海量流也面临着传统存储和单项处理时间的限制。在如此严格的限制下，监测必然是近似的。本研究项目开发了在所有这些约束的积累下对分布式流执行基本监测任务的原则方法。特别是，开发了新的方法来权衡分析的准确性，以满足通信、空间和时间的限制。大量数据流的分布式监控出现在许多通信系统中，主要是在安全应用中。由此产生的模型和解决方案处理这些应用程序，并更好地理解如何在现有资源约束下执行详细的数据分析。这项研究是与朗讯的行业研究人员（Minos Garofalakis和Rajeev Rastogi）合作进行的，他们带来了广泛的流数据挖掘知识，并为测试近似分布式流跟踪的新算法提供了数据集。此外，该项目的工业参与通过技术转让增加了该项目的影响。联合研究该问题的算法、数据库和网络方面将带来重要的新见解和培训。解决方案和由此产生的软件程序将通过该项目的网站（http://www.cs.rutgers.edu/~muthu/adst.html）免费提供。

This research develops models, algorithmic methods and software solutions for tracking of massive data streams for monitoring applications such as IP (Internet Protocol) network traffic analysis. Such monitoring applications are inherently distributed, relying on correlating multiple streams, and therefore present challenges in terms of severe communication constraints. In addition, such massive streams are also faced with the traditional storage and per-item processing time constraints even in the centralized or the single stream cases. Under such severe constraints, monitoring is necessarily approximate. This research project develops principled methods for performing essential monitoring tasks on distributed streams under the accumulation of all such constraints. In particular, new methods are developed that trade off accuracy of analysis for meeting communication, space and time constraints. Distributed monitoring of massive data streams arises in many communication systems, primarily in security applications. The resulting models and solutions address such applications and yield better understanding of how to perform detailed data analyses within existing resource constraints. This research is carried out in collaboration with industry researchers (Minos Garofalakis and Rajeev Rastogi of Lucent) who bring extensive knowledge of stream data mining and provide data sets for testing the new algorithms for approximate distributed stream tracking. In addition, the industrial participation in this project increases the impact of this project via technology transfer. Studying the algorithmic, database and networking aspects of the problem jointly will lead to significant new insights and training. Solutions and resulting software programs will be made freely available via the project's Web site (http://www.cs.rutgers.edu/~muthu/adst.html).