Optimizing hadoop to scale to big systems and big-data

整个大数据市场的价值预计将超过1000亿美元，增长速度大约是整个软件业务的两倍[德勤]。社交媒体等“大数据”的出现，在商业分析中发挥了关键作用。正如IDC的一份报告指出的那样，数据分析将在加拿大城市(例如多伦多)和医疗保健的交通管理中发挥关键作用。这样的大数据通常存储在新一代NoSQL数据库中，并由基于MapReduce/Hadoop的大规模并行基础设施处理。该行业正面临着一个至关重要的挑战：数百万现有的基于SQL的成功应用程序能否利用大数据基础设施？许多关键挑战依然存在，包括最佳数据存储格式的设计和数据存储节点之间数据的有效通信。在这个项目中，我们将与我们的合作伙伴辛巴合作，辛巴是大数据技术的行业领先者，并从客户的角度指导我们迎接挑战。我们将做出三项具体贡献：1)低延迟：我们将采用ApacheHave(基于开源Hadoop的NoSQL框架)来利用RDMA(远程直接内存访问)网络，并向外扩展以利用机架规模的内存资源。Ii)减少带宽：我们将开发特定于数据的压缩机制，以最大限度地减少计算机架上的数据移动，并提高数据库容量和数据访问速度，以及iii)``灵活性‘：我们将开发一种工具，使ApacheHave能够动态实施模式，并使基于SQL的终端业务分析工具能够与Hadoop使用的NoSQL数据库后端进行交互。我们还计划将我们的更改集成到开源的ApacheHave平台中，以使更广泛的大数据社区受益。

The overall `big-data' market is expected to be worth more than \$100 billion and growing roughly twice as fast as the software business as a whole [Deloitte]. The emergence of "big data", such as social media, has played critical roles in business analytics. As an IDC report indicates data analysis will play a key role in managing traffic in canadian cities (e.g., Toronto) and healthcare. Such big data is often stored in the new generation of NoSQL databases, and processed by massively parallel MapReduce/Hadoop based infrastructure. The industryis facing an essential challenge: can millions of existing successful applications based on SQL take the advantage of big data infrastructure? Many critical challenges remain including the design of optimal data storage formats and efficient communication of data between data storage nodes. In this project, we will be working with our partner, Simba, an industry leader of big data technology and has guided us towards the challenges that forsee from their client's perspective. We will be making three specific contributions i) Low-latency: We will be adapting Apache Hive (open source Hadoop-based NoSQL framework) to take advantage of RDMA (Remote Direct Memory Accesses) networks and scale out to take advantage of rack scale memory resources. ii) Reducing Bandwidth: We will be developing data-specific compression mechanisms to minimize the data movement across the compute rack and enable both an increase in database volume and data access speed, and iii) ``Flexibility": We will be developing a tool for enabling Apache Hive to dynamically enforce schemas and enable end-client SQL-based business analytic tools to interact with NoSQL database backends employed by Hadoop.We also plan to integrate our changes into the open source Apache Hive platform to benefit the wider Big data community.