Optimizing hadoop to scale to big systems and big-data

点击翻译按钮获取中文摘要

The overall `big-data' market is expected to be worth more than \$100 billion and growing roughly twice as fast as the software business as a whole [Deloitte]. The emergence of "big data", such as social media, has played critical roles in business analytics. As an IDC report indicates data analysis will play a key role in managing traffic in canadian cities (e.g., Toronto) and healthcare. Such big data is often stored in the new generation of NoSQL databases, and processed by massively parallel MapReduce/Hadoop based infrastructure. The industryis facing an essential challenge: can millions of existing successful applications based on SQL take the advantage of big data infrastructure? Many critical challenges remain including the design of optimal data storage formats and efficient communication of data between data storage nodes. In this project, we will be working with our partner, Simba, an industry leader of big data technology and has guided us towards the challenges that forsee from their client's perspective. We will be making three specific contributions i) Low-latency: We will be adapting Apache Hive (open source Hadoop-based NoSQL framework) to take advantage of RDMA (Remote Direct Memory Accesses) networks and scale out to take advantage of rack scale memory resources. ii) Reducing Bandwidth: We will be developing data-specific compression mechanisms to minimize the data movement across the compute rack and enable both an increase in database volume and data access speed, and iii) ``Flexibility": We will be developing a tool for enabling Apache Hive to dynamically enforce schemas and enable end-client SQL-based business analytic tools to interact with NoSQL database backends employed by Hadoop.We also plan to integrate our changes into the open source Apache Hive platform to benefit the wider Big data community.