eHive-RPC: A Remote Procedure Call Public Interface for eHive

eHive-RPC：eHive 的远程过程调用公共接口

• 批准号: BB/M020398/1
• 负责人: Paul Flicek
• 金额: $ 16.82万
• 依托单位: European Bioinformatics Institute
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2015
• 资助国家: 英国
• 起止时间: 2015 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=BB%2FM020398%2F1
• 关键词: eHive; RPC; Remote; Procedure; Call

Ensembl developed 'eHive' as a production system that manages and optimizes the running of tasks (called 'jobs'), on a compute cluster that may have thousands of Central Processing Units (CPUs). A CPU is the hardware within a computer that carries out the instructions of a computer program by performing the basic input and output operations of the system. Each computer has one or more CPUs.Some compute clusters comprise many thousands of CPUs distributed amongst many computers. With so many computers and CPUs, it is important that jobs are sent to these CPUs in a fair and efficient manner, especially when many users are competing to use the same resources. Clusters usually rely on a central queuing system that holds a list of all the jobs that need to be run and can give individual computers in the cluster explicit instructions about which job to execute. This type of queuing system works well if the jobs each take an hour or more to complete. However, when jobs complete faster than they can be scheduled it creates a processing bottleneck e.g. if a job executes in minutes or less. The usual way to solve the bottleneck is to implement another system on top of the scheduler that 'batches' similar jobs together to make operations more efficient.eHive's novel solution to the issue of job queuing is to move away from this central job scheduling: eHive is a 'distributed' processing system based on 'autonomous agents' with the behavioural structure of honeybees, hence the term 'eHive'. eHive maintains the ability to monitor and track jobs via a central 'blackboard'. Workers are efficiently created on a compute cluster, known as a meadow, with no specific task assigned to them. Once running, each worker contacts the blackboard, is able to find the most suitable kind of job, specializes to claim work and runs multiple jobs of this type in a row. Workers are able to re-specialize to claim other types of jobs once they exhaust their original designation. Each worker regularly updates its status in the blackboard to allow other workers to optimize the overall job distribution.The benefits of eHive are (a) a reduction in the overhead of individual job processing, (b) an increase in the maximum number of tasks that can be running at any one time, (c) an increase in the tolerance to faults in the compute cluster, and (d) the allowance of complicated processes running in parallel.Although eHive was originally designed for the purpose of Ensembl, its functionality is applicable to all data types that have large compute requirements. In this project we aim to transform the possibilities of eHive further, by developing a 'Remote Procedure Call system (RPC) for eHive. This will allow jobs to run on remote clusters as well as local clusters, thereby expanding the use of eHive to multiple compute clusters and cloud computing services. This will enable wider use of eHive within data-intensive fields in the life sciences and beyond.

Ensembl开发的“eHive”作为一个生产系统，管理和优化任务（称为“作业”）的运行，在一个计算集群上，可能有数千个中央处理单元（cpu）。中央处理器是计算机内的硬件，它通过执行系统的基本输入和输出操作来执行计算机程序的指令。每台计算机都有一个或多个cpu。一些计算集群由分布在许多计算机中的数千个cpu组成。由于有如此多的计算机和cpu，以公平和有效的方式将作业发送到这些cpu非常重要，特别是当许多用户竞争使用相同的资源时。集群通常依赖于一个中央排队系统，该系统保存着需要运行的所有作业的列表，并且可以向集群中的单个计算机提供关于执行哪个作业的明确指令。如果每个工作需要一个小时或更长时间才能完成，这种排队系统就会很好地工作。然而，当作业完成的速度超过它们可以调度的速度时，就会产生处理瓶颈，例如，如果作业在几分钟或更短的时间内执行。解决瓶颈的通常方法是在调度程序之上实现另一个系统，该系统将类似的作业“批处理”在一起，以提高操作效率。eHive对作业排队问题的新颖解决方案是远离这种中央作业调度：eHive是一个基于具有蜜蜂行为结构的“自主代理”的“分布式”处理系统，因此称为“eHive”。eHive保持了通过中央“黑板”监控和跟踪作业的能力。工人被高效地创建在一个计算集群上，也就是所谓的“草地”上，没有特定的任务分配给他们。一旦运行，每个工人接触黑板，能够找到最合适的工作类型，专门申请工作，并连续运行该类型的多个工作。工人们可以重新专门要求其他类型的工作，一旦他们用尽了他们原来的指定。每个工人定期在黑板上更新自己的状态，以允许其他工人优化整体工作分配。eHive的好处是：(a)减少了单个作业处理的开销，(b)增加了任何时间可以运行的最大任务数量，(c)增加了计算集群对故障的容错性，(d)允许复杂进程并行运行。虽然eHive最初是为Ensembl设计的，但它的功能适用于所有有大量计算需求的数据类型。在这个项目中，我们的目标是通过为eHive开发一个远程过程调用系统（RPC）来进一步改变eHive的可能性。这将允许作业在远程集群和本地集群上运行，从而将eHive的使用扩展到多个计算集群和云计算服务。这将使eHive在生命科学等数据密集型领域得到更广泛的应用。

项目成果

Ensembl 2021.

• DOI: 10.1093/nar/gkaa942
• 发表时间: 2021-01-08
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Howe KL;Achuthan P;Allen J;Allen J;Alvarez-Jarreta J;Amode MR;Armean IM;Azov AG;Bennett R;Bhai J;Billis K;Boddu S;Charkhchi M;Cummins C;Da Rin Fioretto L;Davidson C;Dodiya K;El Houdaigui B;Fatima R;Gall A;Garcia Giron C;Grego T;Guijarro-Clarke C;Haggerty L;Hemrom A;Hourlier T;Izuogu OG;Juettemann T;Kaikala V;Kay M;Lavidas I;Le T;Lemos D;Gonzalez Martinez J;Marugán JC;Maurel T;McMahon AC;Mohanan S;Moore B;Muffato M;Oheh DN;Paraschas D;Parker A;Parton A;Prosovetskaia I;Sakthivel MP;Salam AIA;Schmitt BM;Schuilenburg H;Sheppard D;Steed E;Szpak M;Szuba M;Taylor K;Thormann A;Threadgold G;Walts B;Winterbottom A;Chakiachvili M;Chaubal A;De Silva N;Flint B;Frankish A;Hunt SE;IIsley GR;Langridge N;Loveland JE;Martin FJ;Mudge JM;Morales J;Perry E;Ruffier M;Tate J;Thybert D;Trevanion SJ;Cunningham F;Yates AD;Zerbino DR;Flicek P
• 通讯作者: Flicek P

Ensembl 2022.

• DOI: 10.1093/nar/gkab1049
• 发表时间: 2022-01-07
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Cunningham F;Allen JE;Allen J;Alvarez-Jarreta J;Amode MR;Armean IM;Austine-Orimoloye O;Azov AG;Barnes I;Bennett R;Berry A;Bhai J;Bignell A;Billis K;Boddu S;Brooks L;Charkhchi M;Cummins C;Da Rin Fioretto L;Davidson C;Dodiya K;Donaldson S;El Houdaigui B;El Naboulsi T;Fatima R;Giron CG;Genez T;Martinez JG;Guijarro-Clarke C;Gymer A;Hardy M;Hollis Z;Hourlier T;Hunt T;Juettemann T;Kaikala V;Kay M;Lavidas I;Le T;Lemos D;Marugán JC;Mohanan S;Mushtaq A;Naven M;Ogeh DN;Parker A;Parton A;Perry M;Piližota I;Prosovetskaia I;Sakthivel MP;Salam AIA;Schmitt BM;Schuilenburg H;Sheppard D;Pérez-Silva JG;Stark W;Steed E;Sutinen K;Sukumaran R;Sumathipala D;Suner MM;Szpak M;Thormann A;Tricomi FF;Urbina-Gómez D;Veidenberg A;Walsh TA;Walts B;Willhoft N;Winterbottom A;Wass E;Chakiachvili M;Flint B;Frankish A;Giorgetti S;Haggerty L;Hunt SE;IIsley GR;Loveland JE;Martin FJ;Moore B;Mudge JM;Muffato M;Perry E;Ruffier M;Tate J;Thybert D;Trevanion SJ;Dyer S;Harrison PW;Howe KL;Yates AD;Zerbino DR;Flicek P
• 通讯作者: Flicek P

Ensembl 2017.

• DOI: 10.1093/nar/gkw1104
• 发表时间: 2017-01-04
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Aken BL;Achuthan P;Akanni W;Amode MR;Bernsdorff F;Bhai J;Billis K;Carvalho-Silva D;Cummins C;Clapham P;Gil L;Girón CG;Gordon L;Hourlier T;Hunt SE;Janacek SH;Juettemann T;Keenan S;Laird MR;Lavidas I;Maurel T;McLaren W;Moore B;Murphy DN;Nag R;Newman V;Nuhn M;Ong CK;Parker A;Patricio M;Riat HS;Sheppard D;Sparrow H;Taylor K;Thormann A;Vullo A;Walts B;Wilder SP;Zadissa A;Kostadima M;Martin FJ;Muffato M;Perry E;Ruffier M;Staines DM;Trevanion SJ;Cunningham F;Yates A;Zerbino DR;Flicek P
• 通讯作者: Flicek P

Ensembl 2018.

• DOI: 10.1093/nar/gkx1098
• 发表时间: 2018-01-04
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Zerbino DR;Achuthan P;Akanni W;Amode MR;Barrell D;Bhai J;Billis K;Cummins C;Gall A;Girón CG;Gil L;Gordon L;Haggerty L;Haskell E;Hourlier T;Izuogu OG;Janacek SH;Juettemann T;To JK;Laird MR;Lavidas I;Liu Z;Loveland JE;Maurel T;McLaren W;Moore B;Mudge J;Murphy DN;Newman V;Nuhn M;Ogeh D;Ong CK;Parker A;Patricio M;Riat HS;Schuilenburg H;Sheppard D;Sparrow H;Taylor K;Thormann A;Vullo A;Walts B;Zadissa A;Frankish A;Hunt SE;Kostadima M;Langridge N;Martin FJ;Muffato M;Perry E;Ruffier M;Staines DM;Trevanion SJ;Aken BL;Cunningham F;Yates A;Flicek P
• 通讯作者: Flicek P

Ensembl 2016.

• DOI: 10.1093/nar/gkv1157
• 发表时间: 2016-01-04
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Yates A;Akanni W;Amode MR;Barrell D;Billis K;Carvalho-Silva D;Cummins C;Clapham P;Fitzgerald S;Gil L;Girón CG;Gordon L;Hourlier T;Hunt SE;Janacek SH;Johnson N;Juettemann T;Keenan S;Lavidas I;Martin FJ;Maurel T;McLaren W;Murphy DN;Nag R;Nuhn M;Parker A;Patricio M;Pignatelli M;Rahtz M;Riat HS;Sheppard D;Taylor K;Thormann A;Vullo A;Wilder SP;Zadissa A;Birney E;Harrow J;Muffato M;Perry E;Ruffier M;Spudich G;Trevanion SJ;Cunningham F;Aken BL;Zerbino DR;Flicek P
• 通讯作者: Flicek P