Optimizing compute task scheduling at Shopify

大数据处理框架允许企业基于从近乎实时的数据中提取的关键见解来驱动业务。Apache Spark是一个框架，Shopify利用它来收集和分析客户信息，并向作为Shopify平台用户的店主公开数据见解。例如，使用Shopify平台建立在线商店的企业可以向该平台发送预先设置的查询，以了解其最近的营销活动（例如，改进b谷歌广告关键字）按地理区域或年龄组的影响，并查看营销活动是否导致实际产品销售。Shopify使用Spark通过处理管道传递数据来分析数据。管道由多个作业组成，每个作业对数据应用一些计算。一组连接的作业形成一个流，该流可能对应于向商店所有者公开的查询。在这个项目中，我们将探索算法解决方案，在吸收作业失败引起的变化的同时，保持有用的流执行频率，并确保在没有资源短缺的情况下，预先安排和临时作业的共同执行。我们打算应用基于启发式和约束满足的方法，研究它们的权衡，并开发一个在实践中最有效的解决方案。

Big Data processing frameworks allow enterprises to drive their business based on key insights extracted fromnear real-time data. Apache Spark is a framework that Shopify leverages to collect and analyze customerinformation and expose data insights to the shop owners who are users of the Shopify platform. For example,the enterprise which built its online store using the Shopify platform can send pre-set queries to the platform tolearn about the impact of their recent marketing campaign (e.g., with an improved Google ad keywords) bygeographical region or by age group, and see if the marketing campaign resulted in actual product sales.Shopify uses Spark to analyze data by passing the data through a processing pipeline. The pipeline consists ofmultiple jobs, each applying some computation to the data. A set of connected jobs form a flow which mightcorrespond to the query exposed to the shop owner. In this project we will explore algorithmic solutions tomaintain a useful flow execution frequency while absorbing the variance caused by job failures, and ensuringco-execution of pre-scheduled and ad-hoc jobs without resource starvation. We intend to apply approachesbased on heuristics and constraint satisfaction, study their tradeoffs, and develop a solution that works best inpractice.