Online Mining of Big Data Streams Using Cloud Computing

点击翻译按钮获取中文摘要

In a world where data are growing at extraordinary rates, there is huge demand for fast and effective analysis of big data to discover useful information for making business decisions. This research program tackles the problem of discovering useful information from big data streams. Big data streams, characterized by high volume and high velocity, have become ubiquitous as many sources (such as social networks, sensor networks and financial markets) produce data continuously and rapidly. Effectively and efficiently discovering patterns from such massive and fast-evolving data will allow businesses to quickly react to their dynamically changing environment to, for example, perform fraud detection at a point of sale, determine which ad to show, or detect spam in comments on news in which trends change quickly in time. Many challenges exist in discovering useful information from big data streams. To handle very fast data, systems have to process the data as fast as the arriving data. However, most existing data stream mining methods are sequential algorithms that run on a single machine and are limited by the memory and speed of the machine. To mine massive data, parallel and distributed computing over a cloud of computers has become a mainstream solution to achieve low latency and high scalability, and MapReduce has become a popular programming paradigm for easily writing applications that process massive data in parallel in a fault-tolerant manner. However, converting a stream mining algorithm into an online parallel MapReduce-style algorithm poses challenges. Most learning algorithms are highly sequential. Parallelizing such algorithms needs considerable efforts and may require the design of new algorithms. In addition, in stream environments, data flow into the system at a rate over which we have no control. The processing system must keep up with the data rate or degrade gracefully. Resource adaptive online learning with bounded approximation is highly needed, which has not been addressed adequately in the MapReduce-style data processing model.To address the above challenges, we will develop parallel versions of stream-mining algorithms using MapReduce-style distributed stream-processing platforms. We will build on our previous and on-going research in data mining and parallelize the stream-mining algorithms that we have developed recently, which include, but not limited to, classification rule learning, high utility pattern mining, and Monte Carlo based learning algorithms. In addition, we will develop resource-adaptive techniques for learning from big data streams. Adaptive data structures and anytime learning algorithms will be developed that can produce best possible answers under resource constraints and can utilize the extra time and memory, if given, to increase the quality of the answers. Moreover, we will identify the pros and cons in developing parallel stream mining algorithms using the state-of-the-art MapReduce-style stream processing platforms and provide feedbacks to the community as to what is further needed in these platforms for them to better serve online learning of big data streams in the cloud.Mining big and fast data streams using cloud computing is still in its infancy. The proposed research will advance the field by proposing novel solutions to its open challenges and will have a wide range of applications in various fields that produce massive data streams.