Optimizing hadoop to scale to big systems and big-data

优化 hadoop 以扩展到大系统和大数据

• 批准号: 485325-2015
• 负责人: Shriraman, Arrvindh
• 金额: $ 5.77万
• 依托单位: Simon Fraser University
• 依托单位国家: 加拿大
• 项目类别: Collaborative Research and Development Grants
• 财政年份: 2017
• 资助国家: 加拿大
• 起止时间: 2017-01-01 至 2018-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=618851
• 关键词: Optimizing; hadoop; scale; big; systems

The overall `big-data' market is expected to be worth more than \$100 billion and growing roughly twice as fast as the software business as a whole [Deloitte]. The emergence of "big data", such as social media, has played critical roles in business analytics. As an IDC report indicates data analysis will play a key role in managing traffic in canadian cities (e.g., Toronto) and healthcare. Such big data is often stored in the new generation of NoSQL databases, and processed by massively parallel MapReduce/Hadoop based infrastructure. The industryis facing an essential challenge: can millions of existing successful applications based on SQL take the advantage of big data infrastructure? Many critical challenges remain including the design of optimal data storage formats and efficient communication of data between data storage nodes. In this project, we will be working with our partner, Simba, an industry leader of big data technology and has guided us towards the challenges that forsee from their client's perspective. We will be making three specific contributions i) Low-latency: We will be adapting Apache Hive (open source Hadoop-based NoSQL framework) to take advantage of RDMA (Remote Direct Memory Accesses) networks and scale out to take advantage of rack scale memory resources. ii) Reducing Bandwidth: We will be developing data-specific compression mechanisms to minimize the data movement across the compute rack and enable both an increase in database volume and data access speed, and iii) ``Flexibility": We will be developing a tool for enabling Apache Hive to dynamically enforce schemas and enable end-client SQL-based business analytic tools to interact with NoSQL database backends employed by Hadoop.We also plan to integrate our changes into the open source Apache Hive platform to benefit the wider Big data community.

整个“大数据”市场的价值预计将超过1000亿美元，增长速度大约是整个软件行业的两倍[德勤]。社交媒体等“大数据”的出现在商业分析中发挥了关键作用。IDC的一份报告显示，数据分析将在加拿大城市的交通管理中发挥关键作用（例如，多伦多）和医疗保健。这些大数据通常存储在新一代NoSQL数据库中，并由基于MapReduce/Hadoop的大规模并行基础设施处理。行业正面临着一个重要的挑战：数百万现有的基于SQL的成功应用程序能否利用大数据基础设施？许多关键挑战仍然存在，包括最佳数据存储格式的设计和数据存储节点之间的高效数据通信。在这个项目中，我们将与我们的合作伙伴Simba合作，Simba是大数据技术的行业领导者，并从客户的角度指导我们应对挑战。我们将做出三个具体的贡献i）低延迟：我们将调整Apache Hive（基于Hadoop的开源NoSQL框架），以利用RDMA（远程直接内存访问）网络，并向外扩展以利用机架规模的内存资源。2.减少带宽：我们将开发数据特定的压缩机制，以尽量减少计算机架上的数据移动，并增加数据库容量和数据访问速度，以及iii）"灵活性”：我们将开发一个工具，使Apache Hive能够动态实施模式并启用终端客户端SQL-我们还计划将我们的变化集成到开源Apache Hive平台中，以使更广泛的人受益。大数据社区