eHive-RPC: A Remote Procedure Call Public Interface for eHive

EnSembl开发了“eHave”作为一个生产系统，用于在可能具有数千个中央处理器(CPU)的计算集群上管理和优化任务(称为“作业”)的运行。CPU是计算机中的硬件，它通过执行系统的基本输入和输出操作来执行计算机程序的指令。每台计算机都有一个或多个CPU。一些计算集群由分布在多台计算机中的数千个CPU组成。由于有如此多的计算机和CPU，以公平和高效的方式将作业发送到这些CPU非常重要，尤其是在许多用户争相使用相同资源的情况下。集群通常依赖于中央排队系统，该系统保存需要运行的所有作业的列表，并且可以向集群中的各个计算机提供关于要执行哪些作业的明确指令。如果每个作业都需要一个小时或更长时间才能完成，则这种类型的排队系统工作得很好。然而，当作业完成的速度快于计划的时间时，就会造成处理瓶颈，例如，如果作业在几分钟或更短的时间内执行。解决瓶颈的通常方法是在调度器之上实现另一个系统，该系统将相似的作业集中在一起以提高操作效率。eHave对作业排队问题的新解决方案是摆脱这种中央作业调度：eHave是一个基于具有蜜蜂行为结构的自主代理的分布式处理系统，因此被称为eHave。EHave通过一个中央“黑板”维护监控和跟踪作业的能力。工作人员是在称为Meadow的计算集群上高效创建的，没有分配给他们特定的任务。一旦运行，每个工人接触黑板，就能找到最合适的工作类型，专门认领工作，并连续运行多个这种类型的工作。一旦用完了最初的称谓，工人们就可以重新专业化地申请其他类型的工作。每个工作器定期更新其在黑板中的状态，以允许其他工作器优化整个作业分配。eHave的好处是(A)减少单个作业处理的开销，(B)增加可同时运行的最大任务数，(C)增加计算集群中的容错能力，以及(D)允许复杂进程并行运行。尽管eHave最初是为EnSembl的目的而设计的，但其功能适用于具有大量计算要求的所有数据类型。在这个项目中，我们的目标是通过为eHave开发一个远程过程调用系统(RPC)，进一步改变eHave的可能性。这将允许作业在远程集群和本地集群上运行，从而将eHave的使用扩展到多个计算集群和云计算服务。这将使eHave在生命科学等数据密集型领域得到更广泛的使用。

Ensembl developed 'eHive' as a production system that manages and optimizes the running of tasks (called 'jobs'), on a compute cluster that may have thousands of Central Processing Units (CPUs). A CPU is the hardware within a computer that carries out the instructions of a computer program by performing the basic input and output operations of the system. Each computer has one or more CPUs.Some compute clusters comprise many thousands of CPUs distributed amongst many computers. With so many computers and CPUs, it is important that jobs are sent to these CPUs in a fair and efficient manner, especially when many users are competing to use the same resources. Clusters usually rely on a central queuing system that holds a list of all the jobs that need to be run and can give individual computers in the cluster explicit instructions about which job to execute. This type of queuing system works well if the jobs each take an hour or more to complete. However, when jobs complete faster than they can be scheduled it creates a processing bottleneck e.g. if a job executes in minutes or less. The usual way to solve the bottleneck is to implement another system on top of the scheduler that 'batches' similar jobs together to make operations more efficient.eHive's novel solution to the issue of job queuing is to move away from this central job scheduling: eHive is a 'distributed' processing system based on 'autonomous agents' with the behavioural structure of honeybees, hence the term 'eHive'. eHive maintains the ability to monitor and track jobs via a central 'blackboard'. Workers are efficiently created on a compute cluster, known as a meadow, with no specific task assigned to them. Once running, each worker contacts the blackboard, is able to find the most suitable kind of job, specializes to claim work and runs multiple jobs of this type in a row. Workers are able to re-specialize to claim other types of jobs once they exhaust their original designation. Each worker regularly updates its status in the blackboard to allow other workers to optimize the overall job distribution.The benefits of eHive are (a) a reduction in the overhead of individual job processing, (b) an increase in the maximum number of tasks that can be running at any one time, (c) an increase in the tolerance to faults in the compute cluster, and (d) the allowance of complicated processes running in parallel.Although eHive was originally designed for the purpose of Ensembl, its functionality is applicable to all data types that have large compute requirements. In this project we aim to transform the possibilities of eHive further, by developing a 'Remote Procedure Call system (RPC) for eHive. This will allow jobs to run on remote clusters as well as local clusters, thereby expanding the use of eHive to multiple compute clusters and cloud computing services. This will enable wider use of eHive within data-intensive fields in the life sciences and beyond.