eHive-RPC: A Remote Procedure Call Public Interface for eHive

点击翻译按钮获取中文摘要

Ensembl developed 'eHive' as a production system that manages and optimizes the running of tasks (called 'jobs'), on a compute cluster that may have thousands of Central Processing Units (CPUs). A CPU is the hardware within a computer that carries out the instructions of a computer program by performing the basic input and output operations of the system. Each computer has one or more CPUs.Some compute clusters comprise many thousands of CPUs distributed amongst many computers. With so many computers and CPUs, it is important that jobs are sent to these CPUs in a fair and efficient manner, especially when many users are competing to use the same resources. Clusters usually rely on a central queuing system that holds a list of all the jobs that need to be run and can give individual computers in the cluster explicit instructions about which job to execute. This type of queuing system works well if the jobs each take an hour or more to complete. However, when jobs complete faster than they can be scheduled it creates a processing bottleneck e.g. if a job executes in minutes or less. The usual way to solve the bottleneck is to implement another system on top of the scheduler that 'batches' similar jobs together to make operations more efficient.eHive's novel solution to the issue of job queuing is to move away from this central job scheduling: eHive is a 'distributed' processing system based on 'autonomous agents' with the behavioural structure of honeybees, hence the term 'eHive'. eHive maintains the ability to monitor and track jobs via a central 'blackboard'. Workers are efficiently created on a compute cluster, known as a meadow, with no specific task assigned to them. Once running, each worker contacts the blackboard, is able to find the most suitable kind of job, specializes to claim work and runs multiple jobs of this type in a row. Workers are able to re-specialize to claim other types of jobs once they exhaust their original designation. Each worker regularly updates its status in the blackboard to allow other workers to optimize the overall job distribution.The benefits of eHive are (a) a reduction in the overhead of individual job processing, (b) an increase in the maximum number of tasks that can be running at any one time, (c) an increase in the tolerance to faults in the compute cluster, and (d) the allowance of complicated processes running in parallel.Although eHive was originally designed for the purpose of Ensembl, its functionality is applicable to all data types that have large compute requirements. In this project we aim to transform the possibilities of eHive further, by developing a 'Remote Procedure Call system (RPC) for eHive. This will allow jobs to run on remote clusters as well as local clusters, thereby expanding the use of eHive to multiple compute clusters and cloud computing services. This will enable wider use of eHive within data-intensive fields in the life sciences and beyond.