Optimizing compute task scheduling at Shopify

点击翻译按钮获取中文摘要

Big Data processing frameworks allow enterprises to drive their business based on key insights extracted fromnear real-time data. Apache Spark is a framework that Shopify leverages to collect and analyze customerinformation and expose data insights to the shop owners who are users of the Shopify platform. For example,the enterprise which built its online store using the Shopify platform can send pre-set queries to the platform tolearn about the impact of their recent marketing campaign (e.g., with an improved Google ad keywords) bygeographical region or by age group, and see if the marketing campaign resulted in actual product sales.Shopify uses Spark to analyze data by passing the data through a processing pipeline. The pipeline consists ofmultiple jobs, each applying some computation to the data. A set of connected jobs form a flow which mightcorrespond to the query exposed to the shop owner. In this project we will explore algorithmic solutions tomaintain a useful flow execution frequency while absorbing the variance caused by job failures, and ensuringco-execution of pre-scheduled and ad-hoc jobs without resource starvation. We intend to apply approachesbased on heuristics and constraint satisfaction, study their tradeoffs, and develop a solution that works best inpractice.