Scaling big data mining infrastructure: the twitter experience
Scaling big data mining infrastructure: the twitter experience
复制标题
DOI：
10.1145/2481244.2481247
复制
发表时间：
2013-04
期刊：
影响因子：
--
通讯作者：
Jimmy J. Lin;D. Ryaboy
中科院分区：
文献类型：
--
作者：
Jimmy J. Lin;D. Ryaboy
在过去的几年里，Twitter的分析平台在规模、复杂性、用户数量和各种用例方面都经历了巨大的增长。在本文中，我们讨论了我们的基础设施的演变和发展的能力，数据挖掘的“大数据”。一个重要的教训是，在实践中成功的大数据挖掘远远超过了大多数学者所认为的数据挖掘：“战壕”中的生活被数据挖掘算法应用之前的大量准备工作所占据，然后是将初步模型转化为强大的解决方案的大量努力。在这种情况下，我们讨论两个主题：首先，模式在帮助数据科学家理解PB级数据存储方面发挥着重要作用，但它们不足以提供可用于生成见解的数据的整体“大局”。其次，我们观察到，构建数据分析平台的一个主要挑战源于必须集成到生产工作流程中的各种组件的异构性-我们称之为“管道”。本文有两个目的：对于从业者，我们希望分享我们的经验，为后来者铺平道路上的颠簸。对于学术研究人员，我们希望为生产环境中的数据挖掘提供更广泛的背景，为未来的工作指出机会。
The analytics platform at Twitter has experienced tremendous growth over the past few years in terms of size, complexity, number of users, and variety of use cases. In this paper, we discuss the evolution of our infrastructure and the development of capabilities for data mining on "big data". One important lesson is that successful big data mining in practice is about much more than what most academics would consider data mining: life "in the trenches" is occupied by much preparatory work that precedes the application of data mining algorithms and followed by substantial effort to turn preliminary models into robust solutions. In this context, we discuss two topics: First, schemas play an important role in helping data scientists understand petabyte-scale data stores, but they're insufficient to provide an overall "big picture" of the data available to generate insights. Second, we observe that a major challenge in building data analytics platforms stems from the heterogeneity of the various components that must be integrated together into production workflows---we refer to this as "plumbing". This paper has two goals: For practitioners, we hope to share our experiences to flatten bumps in the road for those who come after us. For academic researchers, we hope to provide a broader context for data mining in production environments, pointing out opportunities for future work.