Distributed ReliefF-based feature selection in Spark
Distributed ReliefF-based feature selection in Spark
复制标题
DOI：
10.1007/s10115-017-1145-y
复制
发表时间：
2018-10-01
影响因子：
2.7
通讯作者：
de-Marcos, Luis
中科院分区：
文献类型：
--
作者：
Palma-Mendoza, Raul-Jose;Rodriguez, Daniel;de-Marcos, Luis
特征选择是机器学习和数据挖掘领域的一个重要研究领域，去除不相关和冗余的特征通常有助于减少处理数据集所需的工作量，同时保持甚至提高处理算法的精度。然而，为在一台机器上执行而设计的传统算法缺乏可扩展性，无法处理当前大数据时代可用的日益增长的数据量。ReliefF算法是在许多FS应用中成功实现的最重要的算法之一。在这篇文章中，我们提出了一个完全重新设计的流行的ReliefF算法的分布式版本，基于新的Spark集群计算模型，我们称之为DiReliefF。我们在四个公开可用的数据集上测试了我们的建议的有效性，所有这些数据集都有大量的实例，其中两个也具有大量的特征。这些数据集的子集也用于将结果与算法的非分布式实现进行比较。结果表明，非分布式实现无法在没有专用硬件的情况下处理如此大的数据量，而我们的设计可以以可扩展的方式处理这些数据，并具有更好的处理时间和内存使用。
Feature selection (FS) is a key research area in the machine learning and data mining fields; removing irrelevant and redundant features usually helps to reduce the effort required to process a dataset while maintaining or even improving the processing algorithm's accuracy. However, traditional algorithms designed for executing on a single machine lack scalability to deal with the increasing amount of data that have become available in the current Big Data era. ReliefF is one of the most important algorithms successfully implemented in many FS applications. In this paper, we present a completely redesigned distributed version of the popular ReliefF algorithm based on the novel Spark cluster computing model that we have called DiReliefF. The effectiveness of our proposal is tested on four publicly available datasets, all of them with a large number of instances and two of them with also a large number of features. Subsets of these datasets were also used to compare the results to a non-distributed implementation of the algorithm. The results show that the non-distributed implementation is unable to handle such large volumes of data without specialized hardware, while our design can process them in a scalable way with much better processing times and memory usage.