Online Mining of Big Data Streams Using Cloud Computing

在数据以惊人的速度增长的世界中，对快速有效地分析大数据以发现有用的信息以制定业务决策的需求巨大。这个研究项目解决了从大数据流中发现有用信息的问题。随着社交网络、传感器网络、金融市场等众多来源持续快速地产生数据，以海量、高速度为特征的大数据流已经无处不在。从如此庞大和快速发展的数据中有效和高效地发现模式，将使企业能够快速对动态变化的环境做出反应，例如，在销售点执行欺诈检测，确定要显示哪个广告，或者在趋势变化迅速的新闻评论中检测垃圾邮件。从大数据流中发现有用的信息存在许多挑战。为了处理非常快的数据，系统处理数据的速度必须和到达数据的速度一样快。然而，大多数现有的数据流挖掘方法是在单个机器上运行的顺序算法，并且受机器的内存和速度的限制。为了挖掘海量数据，计算机云上的并行和分布式计算已经成为实现低延迟和高可扩展性的主流解决方案，MapReduce已经成为一种流行的编程范式，可以轻松编写以容错方式并行处理海量数据的应用程序。然而，将流挖掘算法转换为在线并行mapreduce风格的算法存在挑战。大多数学习算法都是高度顺序的。并行化这样的算法需要相当大的努力，可能需要设计新的算法。此外，在流环境中，数据以我们无法控制的速率流入系统。处理系统必须跟上数据速率，否则就会优雅地降级。具有有界近似的资源自适应在线学习是非常必要的，这在mapreduce风格的数据处理模型中没有得到充分的解决。为了解决上述挑战，我们将使用mapreduce风格的分布式流处理平台开发并行版本的流挖掘算法。我们将以我们之前和正在进行的数据挖掘研究为基础，并行化我们最近开发的流挖掘算法，包括但不限于分类规则学习、高效用模式挖掘和基于蒙特卡罗的学习算法。此外，我们将开发从大数据流中学习的资源适应技术。将开发自适应数据结构和随时学习算法，这些算法可以在资源限制下产生最佳答案，并且可以利用额外的时间和内存来提高答案的质量。此外，我们将确定使用最先进的mapreduce风格的流处理平台开发并行流挖掘算法的利弊，并向社区提供反馈，说明这些平台还需要什么，以便更好地为云中的大数据流的在线学习提供服务。使用云计算挖掘大而快速的数据流仍处于起步阶段。拟议的研究将通过为其开放挑战提出新颖的解决方案来推进该领域，并将在产生大量数据流的各个领域具有广泛的应用。

In a world where data are growing at extraordinary rates, there is huge demand for fast and effective analysis of big data to discover useful information for making business decisions. This research program tackles the problem of discovering useful information from big data streams. Big data streams, characterized by high volume and high velocity, have become ubiquitous as many sources (such as social networks, sensor networks and financial markets) produce data continuously and rapidly. Effectively and efficiently discovering patterns from such massive and fast-evolving data will allow businesses to quickly react to their dynamically changing environment to, for example, perform fraud detection at a point of sale, determine which ad to show, or detect spam in comments on news in which trends change quickly in time. Many challenges exist in discovering useful information from big data streams. To handle very fast data, systems have to process the data as fast as the arriving data. However, most existing data stream mining methods are sequential algorithms that run on a single machine and are limited by the memory and speed of the machine. To mine massive data, parallel and distributed computing over a cloud of computers has become a mainstream solution to achieve low latency and high scalability, and MapReduce has become a popular programming paradigm for easily writing applications that process massive data in parallel in a fault-tolerant manner. However, converting a stream mining algorithm into an online parallel MapReduce-style algorithm poses challenges. Most learning algorithms are highly sequential. Parallelizing such algorithms needs considerable efforts and may require the design of new algorithms. In addition, in stream environments, data flow into the system at a rate over which we have no control. The processing system must keep up with the data rate or degrade gracefully. Resource adaptive online learning with bounded approximation is highly needed, which has not been addressed adequately in the MapReduce-style data processing model.To address the above challenges, we will develop parallel versions of stream-mining algorithms using MapReduce-style distributed stream-processing platforms. We will build on our previous and on-going research in data mining and parallelize the stream-mining algorithms that we have developed recently, which include, but not limited to, classification rule learning, high utility pattern mining, and Monte Carlo based learning algorithms. In addition, we will develop resource-adaptive techniques for learning from big data streams. Adaptive data structures and anytime learning algorithms will be developed that can produce best possible answers under resource constraints and can utilize the extra time and memory, if given, to increase the quality of the answers. Moreover, we will identify the pros and cons in developing parallel stream mining algorithms using the state-of-the-art MapReduce-style stream processing platforms and provide feedbacks to the community as to what is further needed in these platforms for them to better serve online learning of big data streams in the cloud.Mining big and fast data streams using cloud computing is still in its infancy. The proposed research will advance the field by proposing novel solutions to its open challenges and will have a wide range of applications in various fields that produce massive data streams.