Model-Parallel Collaborative Filtering in Apache Spark

Apache Spark 中的模型并行协同过滤

• 批准号: 1555772
• 负责人: Ameet Talwalkar
• 金额: $ 6.88万
• 依托单位: University of California-Los Angeles
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2016-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1555772&HistoricalAwards=false
• 关键词: Model; Parallel; Collaborative; Filtering; Apache

With data rapidly growing in size and complexity, many organizations are eager to train collaborative filtering methods on massive datasets using distributed computing environments. For instance, Netflix has hundreds of thousands of online programs to recommend to its millions of users, and Facebook has millions of users who could potentially form new links between one another. However, leading methods introduce significant algorithmic challenges in the distributed setting. The PI proposes to study a novel algorithm designed to be efficient for large-scale data science applications. Preliminary studies demonstrate the promise of this method, and the PI proposes to formally characterize the algorithm's behavior, perform an extensive empirical evaluation, and incorporate ideas inspired by this proposal into an upcoming online course PI will be teaching.Collaborative filtering, and in particular matrix factorization, is a widely used method for devising recommender systems. However, the size of these models grows linearly with the number of users and items, and leading methods for matrix factorization introduce significant challenges in the distributed setting due to their high communication costs. The PI proposes to study a novel model-parallel algorithm designed for Apache Spark that leverages the sparsity of the underlying data to drastically reduce this communication burden. Preliminary studies demonstrate the promise of this method, and the PI proposes to formally characterize the algorithm's behavior, perform an extensive empirical evaluation, and explore the paradigm of model-parallelism in Spark more generally for other learning settings. The PI will also incorporate ideas related to model-parallelism inspired by this proposal into an upcoming MOOC that be taught on the edX platform.

随着数据的规模和复杂性的快速增长，许多组织都渴望使用分布式计算环境来训练针对海量数据集的协作过滤方法。例如，Netflix有数十万个在线节目要推荐给它的数百万用户，Facebook有数百万用户可能会在彼此之间形成新的链接。然而，领先的方法在分布式环境中引入了重大的算法挑战。PI建议研究一种新的算法，旨在为大规模数据科学应用而设计的高效算法。初步研究证明了这种方法的前景，PI建议形式化地描述算法的行为，执行广泛的经验评估，并将受此建议启发的想法纳入即将教授的在线课程PI中。协作过滤，特别是矩阵分解，是设计推荐系统的一种广泛使用的方法。然而，这些模型的大小随着用户和项目的数量线性增长，并且由于其高昂的通信成本，领先的矩阵分解方法在分布式环境中带来了巨大的挑战。PI建议研究一种新的模型--为ApacheSpark设计的并行算法，该算法利用底层数据的稀疏性来极大地减少这种通信负担。初步研究证明了这种方法的前景，PI建议对算法的行为进行形式化描述，执行广泛的经验评估，并为其他学习环境探索Spark中更一般的模型并行性范例。PI还将把受此提议启发的与模型并行性相关的想法纳入即将在edX平台上教授的MOOC中。