Scalable and Consistent Management of Large Scale Data

点击翻译按钮获取中文摘要

Effective large scale data management is becoming important for scientific discovery and for maintaining a competitive edge in business and information technology. Large scale data management, sometimes called “Big Data” management, is often used to refer to the volume of data and the rate at which it is generated. Scientific research organizations like CERN and companies such as Twitter, Walmart and Facebook all generate and manage very large amounts of data. For example, starting 2016, the Large Synoptic Survey Telescope will collect 30 terabytes of data every night for 10 years. Over 100 terabytes of data are uploaded daily to Facebook, which has more than 1 billion active users around the globe. To manage these large scale data, massive distribution over multiple, large, storage and processing server clouds both within and across geographically distributed data centers is required. Furthermore, effective management of these server clouds is needed so that operational costs are reduced. To this end, key research problems need to be addressed for large scale data management. First, dynamic database distribution techniques are required both within and across server clouds to deal with latencies contributing to response times. These latencies can be due to, for example, overall changing patterns in workload requests or the emergence of database hot spots as a result of load spikes. Second, with the growth of data centers, there is a need to achieve savings in terms of server maintenance cost of storing and managing large scale data. Thus, techniques to consolidate database server loads become key to reduce costs and keep large scale data management viable. The goal of this proposal is to design and develop a set of protocols, algorithms and systems that will deliver viable solutions to these research problems using dynamic, on-the-fly, large scale data management techniques. These techniques will advance the state-of-the-art in distributed data management and provide systems designers and administrators with "knobs" that can be controlled to vary the degree of dynamism required to reduce the costs of managing the data while quantifying performance trade-offs.