Optimizing compute task scheduling at Shopify

大数据处理框架允许企业基于从近实时数据中提取的关键见解来推动其业务。Apache Spark是Shopify用来收集和分析客户信息的框架，并向Shopify平台的用户提供数据见解。例如，使用Shopify平台建立其在线商店的企业可以向平台发送预设查询，以了解其最近营销活动的影响（例如，使用改进的Google广告关键字），并查看营销活动是否导致实际产品销售。Shopify使用Spark通过处理管道传递数据来分析数据。管道由多个作业组成，每个作业对数据进行一些计算。一组连接的作业形成一个流，该流可能对应于向店主公开的查询。在这个项目中，我们将探索算法解决方案，以保持一个有用的流执行频率，同时吸收由作业失败造成的差异，并确保共同执行的预先安排和ad hoc作业，而不会出现资源饥饿。我们打算应用基于逻辑学和约束满足的方法，研究它们的权衡，并开发出一种在实践中效果最好的解决方案。

Big Data processing frameworks allow enterprises to drive their business based on key insights extracted fromnear real-time data. Apache Spark is a framework that Shopify leverages to collect and analyze customerinformation and expose data insights to the shop owners who are users of the Shopify platform. For example,the enterprise which built its online store using the Shopify platform can send pre-set queries to the platform tolearn about the impact of their recent marketing campaign (e.g., with an improved Google ad keywords) bygeographical region or by age group, and see if the marketing campaign resulted in actual product sales.Shopify uses Spark to analyze data by passing the data through a processing pipeline. The pipeline consists ofmultiple jobs, each applying some computation to the data. A set of connected jobs form a flow which mightcorrespond to the query exposed to the shop owner. In this project we will explore algorithmic solutions tomaintain a useful flow execution frequency while absorbing the variance caused by job failures, and ensuringco-execution of pre-scheduled and ad-hoc jobs without resource starvation. We intend to apply approachesbased on heuristics and constraint satisfaction, study their tradeoffs, and develop a solution that works best inpractice.