PMLB: a large benchmark suite for machine learning evaluation and comparison.
PMLB: a large benchmark suite for machine learning evaluation and comparison.
复制标题
DOI：
10.1186/s13040-017-0154-4
复制
发表时间：
2017
期刊：
影响因子：
4.5
通讯作者：
Moore JH
中科院分区：
文献类型：
--
作者：
Olson RS;La Cava W;Orzechowski P;Urbanowicz RJ;Moore JH
根据特定研究的目标问题和目标，数据挖掘中机器学习方法的选择、开发或比较可能是一项艰巨的任务。许多公开可用的真实世界和模拟基准数据集来自不同的来源，但它们的组织和作为标准的采用并不一致。因此，选择和策划特定基准仍然是机器学习从业者和数据科学家不必要的负担。本研究引入了一个可访问、精心策划和开发的公共基准资源，以帮助识别不同机器学习方法的优点和缺点。我们比较该资源中当前一组基准数据集的元特征，以表征可用数据的多样性。最后，我们将一些已建立的机器学习方法应用于整个基准测试套件，并分析数据集和算法在性能方面如何聚类。从这项研究中，我们发现现有的基准测试缺乏正确基准测试机器学习算法的多样性，并且在基准测试问题上仍然存在一些差距需要考虑。这项工作代表了理解流行基准测试套件的局限性和开发将现有基准测试标准与未来更多样化和更高效的标准连接起来的资源的又一个重要步骤。
The selection, development, or comparison of machine learning methods in data mining can be a difficult task based on the target problem and goals of a particular study. Numerous publicly available real-world and simulated benchmark datasets have emerged from different sources, but their organization and adoption as standards have been inconsistent. As such, selecting and curating specific benchmarks remains an unnecessary burden on machine learning practitioners and data scientists. The present study introduces an accessible, curated, and developing public benchmark resource to facilitate identification of the strengths and weaknesses of different machine learning methodologies. We compare meta-features among the current set of benchmark datasets in this resource to characterize the diversity of available data. Finally, we apply a number of established machine learning methods to the entire benchmark suite and analyze how datasets and algorithms cluster in terms of performance. From this study, we find that existing benchmarks lack the diversity to properly benchmark machine learning algorithms, and there are several gaps in benchmarking problems that still need to be considered. This work represents another important step towards understanding the limitations of popular benchmarking suites and developing a resource that connects existing benchmarking standards to more diverse and efficient standards in the future.
登录
查看更多内容
影响因子：
4.5
作者：
Urbanowicz RJ;Kiralis J;Sinnott-Armstrong NA;Heberling T;Fisher JM;Moore JH
通讯作者：
Moore JH
影响因子：
2.2
作者：
KOZA, JR
通讯作者：
KOZA, JR
影响因子：
2.6
作者：
White, David R.;McDermott, James;Luke, Sean
通讯作者：
Luke, Sean
DOI：
10.1016/j.engappai.2016.07.004
发表时间：
2016-10-01
影响因子：
8
作者：
La Cava, William;Danai, Kourosh;Spector, Lee
通讯作者：
Spector, Lee
影响因子：
2.6
作者：
Urbanowicz RJ;Moore JH
通讯作者：
Moore JH