Optimizing hadoop to scale to big systems and big-data

整个“大数据”市场的价值预计将超过1000亿美元，增长速度大约是整个软件业务的两倍[德勤]。社交媒体等“大数据”的出现，在商业分析中发挥了关键作用。IDC的一份报告指出，数据分析将在管理加拿大城市（如多伦多）的交通和医疗保健方面发挥关键作用。这样的大数据通常存储在新一代的NoSQL数据库中，并通过基于MapReduce/Hadoop的大规模并行基础设施进行处理。这个行业正面临着一个重要的挑战：数百万基于SQL的成功应用程序能否利用大数据基础设施的优势？许多关键的挑战仍然存在，包括设计最佳的数据存储格式和数据存储节点之间的有效数据通信。在这个项目中，我们将与我们的合作伙伴Simba合作，Simba是大数据技术的行业领导者，并从客户的角度指导我们应对挑战。我们将做出三个具体的贡献：1)低延迟：我们将调整Apache Hive（基于hadoop的开源NoSQL框架）来利用RDMA（远程直接内存访问）网络，并向外扩展以利用机架级内存资源。ii)减少带宽：我们将开发特定于数据的压缩机制，以最大限度地减少数据在计算机架上的移动，从而增加数据库容量和数据访问速度。iii)“灵活性”：我们将开发一个工具，使Apache Hive能够动态执行模式，并使基于sql的终端客户端业务分析工具能够与Hadoop使用的NoSQL数据库后端进行交互。我们还计划将我们的变更集成到开源Apache Hive平台中，以使更广泛的大数据社区受益。

The overall `big-data' market is expected to be worth more than \$100 billion and growing roughly twice as fast as the software business as a whole [Deloitte]. The emergence of "big data", such as social media, has played critical roles in business analytics. As an IDC report indicates data analysis will play a key role in managing traffic in canadian cities (e.g., Toronto) and healthcare. Such big data is often stored in the new generation of NoSQL databases, and processed by massively parallel MapReduce/Hadoop based infrastructure. The industryis facing an essential challenge: can millions of existing successful applications based on SQL take the advantage of big data infrastructure? Many critical challenges remain including the design of optimal data storage formats and efficient communication of data between data storage nodes. In this project, we will be working with our partner, Simba, an industry leader of big data technology and has guided us towards the challenges that forsee from their client's perspective. We will be making three specific contributions i) Low-latency: We will be adapting Apache Hive (open source Hadoop-based NoSQL framework) to take advantage of RDMA (Remote Direct Memory Accesses) networks and scale out to take advantage of rack scale memory resources. ii) Reducing Bandwidth: We will be developing data-specific compression mechanisms to minimize the data movement across the compute rack and enable both an increase in database volume and data access speed, and iii) ``Flexibility": We will be developing a tool for enabling Apache Hive to dynamically enforce schemas and enable end-client SQL-based business analytic tools to interact with NoSQL database backends employed by Hadoop.We also plan to integrate our changes into the open source Apache Hive platform to benefit the wider Big data community.