A Database-Hadoop Hybrid Approach to Scalable Machine Learning
A Database-Hadoop Hybrid Approach to Scalable Machine Learning
复制标题
用于可扩展机器学习的数据库-Hadoop 混合方法
DOI：
10.1109/bigdata.congress.2013.10
复制
发表时间：
2013
期刊：
影响因子：
--
通讯作者：
Isao Kojima
中科院分区：
文献类型：
--
作者：
Makoto Yui;Isao Kojima
有两种流行的思想流派用于执行不适合记忆的大规模机器学习。一种是在关系数据库管理系统中运行机器学习，另一种是将分析功能推入MapReduce。由于每种方法都有自己的优点和缺点，我们提出了一种数据库-Hadoop混合方法来进行可扩展的机器学习，其中批量学习在Hadoop平台上执行，而增量学习在PostgreSQL上执行。我们提出了一个纯粹的关系的方法，消除了以前的方法基于用户定义的聚合的可扩展性限制，并讨论了在Hadoop/Hive应用所提出的方法的问题和解决方案。分类性能和训练速度的实验评估是使用KDD Cup 2012，Track 2中提供的商业广告数据集进行的。实验结果表明，与最先进的可扩展机器学习框架相比，我们的方案具有有竞争力的分类性能和优越的上级训练速度，对于回归任务，分别比Vow pal Wabbit和俾斯麦快5倍和7.65倍。
There are two popular schools of thought for performing large-scale machine learning that does not fit into memory. One is to run machine learning within a relational database management system, and the other is to push analytical functions into MapReduce. As each approach has its own set of pros and cons, we propose a database-Hadoop hybrid approach to scalable machine learning where batch-learning is performed on the Hadoop platform, while incremental-learning is performed on PostgreSQL. We propose a purely relational approach that removes the scalability limitation of previous approaches based on user-defined aggregates and also discuss issues and resolutions in applying the proposed approach to Hadoop/Hive. Experimental evaluations of classification performance and training speed were conducted using a commercial advertisement dataset provided in the KDD Cup 2012, Track 2. The experimental results show that our scheme has competitive classification performance and superior training speed compared with state-of-the-art scalable machine learning frameworks, 5 and 7.65 times faster than Vow pal Wabbit and Bismarck, respectively, for a regression task.