MAD Skills: New Analysis Practices for Big Data
MAD Skills: New Analysis Practices for Big Data
复制标题
DOI：
10.14778/1687553.1687576
复制
发表时间：
2009-08-01
影响因子：
2.5
通讯作者：
Welton, Caleb
中科院分区：
文献类型：
--
作者：
Cohen, Jeffrey;Dolan, Brian;Welton, Caleb
随着海量数据的获取和存储变得越来越便宜，各种各样的企业都在雇用统计学家从事复杂的数据分析。在本文中，我们强调了磁性、敏捷、深度（MAD）数据分析的新兴实践，这是对传统企业数据仓库和商业智能的彻底背离。我们展示了我们的设计理念，技术和经验，为世界上最大的广告网络之一的福克斯观众网络提供MAD分析，使用Greenplum并行数据库系统。我们描述了在这些环境中支持分析师敏捷工作风格的数据库设计方法。我们提出了复杂的统计技术的数据并行算法，重点是密度方法。最后，我们讨论了数据库系统的一些特性，这些特性可以通过多种存储机制同时使用SQL和MapReduce接口实现敏捷设计和灵活的算法开发。
As massive data acquisition and storage becomes increasingly affordable, a wide variety of enterprises are employing statisticians to engage in sophisticated data analysis. In this paper we highlight the emerging practice of Magnetic, Agile, Deep (MAD) data analysis as a radical departure from traditional Enterprise Data Warehouses and Business Intelligence. We present our design philosophy, techniques and experience providing MAD analytics for one of the world's largest advertising networks at Fox Audience Network, using the Greenplum parallel database system. We describe database design methodologies that support the agile working style of analysts in these settings. We present dataparallel algorithms for sophisticated statistical techniques, with a focus on density methods. Finally, we reflect on database system features that enable agile design and flexible algorithm development using both SQL and MapReduce interfaces over a variety of storage mechanisms.