Frameworks: Re-Engineering Galaxy for Performance, Scalability and Energy Efficiency

框架：重新设计 Galaxy 以提高性能、可扩展性和能源效率

• 批准号: 1931531
• 负责人: Mahmut Kandemir
• 金额: $ 350万
• 依托单位: Pennsylvania State Univ University Park
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1931531&HistoricalAwards=false
• 关键词: Frameworks; Re; Engineering; Galaxy; Performance

Biomedical research is an important branch of science that deals with the problem of studying biological processes and identifying, preventing and curing diseases. This research forms the pathway to the discovery of new medicines as well as new therapies. As such, biomedical research is crucial to advance the national health and prosperity. Given the geographically distributed research groups and biomedical labs, collaborative science plays a very important role in biomedical research. Galaxy is an open source, web-based framework that is extensively used by more than 20,000 researchers world-wide for conducting research in many application domains, the most prominent of which is biomedical research. It provides a web-based environment using which scientists perform various computational analyses on their data, exchange results from these analyses, explore new research concepts, facilitate student training, and preserve their results for future use. Galaxy currently runs on a large variety of high-performance computing (HPC) platforms including local clusters, supercomputers in national labs, public datacenters and Cloud. Unfortunately, while most of these systems supplement conventional CPUs with significant accelerator capabilities (in the form of Graphical Processing Units (GPUs) and/or Field-Programmable Gate Arrays (FPGAs)), the current Galaxy implementation does not take advantage of these powerful accelerators. This project enhances the Galaxy framework so that it can take full advantage of the tremendous computational capabilities offered by GPUs and FPGAs. By doing so, the important applications running under Galaxy experiences significant speedups, thereby accelerating scientific discoveries. This project consists of four complementary tasks, which follow a logistic progression as follows: Task-I focuses on redesigning existing Galaxy tools with GPU/FPGA support and integrate them to Galaxy tool-chains; Task-II provides containerization support for the tools and accelerator-aware orchestration for running Galaxy on cloud platforms; Task-III implements specific policy driven scheduling schemes for Task-I and Task-II; and finally, Task-IV redesigns Galaxy storage to speed up execution and reduce bottlenecks related to data transfer. The proposed enhancements to Galaxy enables the integration of innovation with discovery by providing a state-of-the art experimental platform to a larger community of researchers across several disciplines. On the broader impact and outreach/educational front, this project impacts the performance and energy efficiency of Galaxy tools and applications and improves the productivity of a typical Galaxy user tremendously; that is, the main beneficiaries of this project are thousands of members of existing Galaxy Community. However, this project also (i) helps existing GPU and FPGA based (non-Galaxy) applications start using Galaxy, thereby taking full advantage of all existing toolsets within the framework, (ii) enables Galaxy tools to take better advantage of emerging cluster scheduling capabilities, and (iii) creates a synergy with concurrent Galaxy related efforts and existing infrastructure efforts the PIs are involved with, to further expedite scientific discoveries. As such, this proposed system support will have a broad societal impact via the enhanced Galaxy system support. On the education side, the project involves under-represented groups in computer science as well as in bio-informatics, outreach to undergraduates, various K-12 related activities (Science-U, CSATS, VIEW), and engagement with researchers in other disciplines (e.g., natural language processing, image processing, drug discovery and cosmology) via a workshop open to the Galaxy community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

生物医学研究是一门重要的科学分支，它研究生物过程，识别、预防和治疗疾病。这项研究形成了发现新药和新疗法的途径。因此，生物医学研究对促进国家健康和繁荣至关重要。由于研究小组和生物医学实验室的地理分布，合作科学在生物医学研究中发挥着非常重要的作用。Galaxy是一个基于Web的开源框架，被全球20，000多名研究人员广泛用于在许多应用领域进行研究，其中最突出的是生物医学研究。 它提供了一个基于网络的环境，科学家可以使用它对数据进行各种计算分析，交换这些分析的结果，探索新的研究概念，促进学生培训，并保存他们的结果供将来使用。Galaxy目前在各种高性能计算（HPC）平台上运行，包括本地集群，国家实验室的超级计算机，公共数据中心和云。不幸的是，虽然这些系统中的大多数都为传统CPU提供了重要的加速器功能（以图形处理单元（GPU）和/或现场可编程门阵列（FPGA）的形式），但当前的Galaxy实现并没有利用这些强大的加速器。该项目增强了Galaxy框架，以便它可以充分利用GPU和FPGA提供的巨大计算能力。通过这样做，在Galaxy下运行的重要应用程序将获得显著的加速，从而加速科学发现。 该项目由四个互补的任务组成，遵循以下逻辑进展：任务I侧重于重新设计具有GPU/FPGA支持的现有Galaxy工具并将其集成到Galaxy工具链中;任务II为工具提供容器化支持，并为在云平台上运行Galaxy提供加速器感知的编排;任务III为任务I和任务II实现特定的策略驱动调度方案;最后，Task-IV重新设计了Galaxy存储，以加快执行速度并减少与数据传输相关的瓶颈。拟议的Galaxy增强功能通过为多个学科的更大研究人员社区提供最先进的实验平台，实现了创新与发现的整合。在更广泛的影响和外联/教育方面，该项目影响到银河工具和应用程序的性能和能效，并极大地提高了银河系统典型用户的生产力;也就是说，该项目的主要受益者是银河系统社区现有的数千名成员。然而，该项目还（i）帮助现有的GPU和FPGA的基础上，（非银河系统）应用程序开始使用银河系统，从而充分利用框架内的所有现有工具集，㈡使银河系统工具能够更好地利用新出现的集群调度能力，㈢与同时进行的银河系统相关工作和主要参与人参与的现有基础设施工作产生协同作用，以进一步加速科学发现。因此，拟议的系统支助将通过加强银河系统支助产生广泛的社会影响。 在教育方面，该项目涉及计算机科学和生物信息学的代表性不足的群体，对本科生的宣传，各种K-12相关活动（Science-U，CSATS，VIEW），以及与其他学科的研究人员的接触（例如，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Kube-Knots: Resource Harvesting through Dynamic Container Orchestration in GPU-based Datacenters

• DOI: 10.1109/cluster.2019.8891040
• 发表时间: 2019-09
• 期刊: 2019 IEEE International Conference on Cluster Computing (CLUSTER)
• 作者: P. Thinakaran;Jashwant Raj Gunasekaran;Bikash Sharma;M. Kandemir;C. Das

Compression Algorithm for Colored de Bruijn Graphs

彩色 de Bruijn 图的压缩算法

• 发表时间: 2023
• 期刊: 23rd International Workshop on Algorithms in Bioinformatics (WABI 2023
• 作者: Rahman, Amatur;Dufresne, Yoann;Medvedev, Paul
• 通讯作者: Medvedev, Paul

Multiverse: Dynamic VM Provisioning for Virtualized High Performance Computing Clusters

• DOI: 10.1109/ccgrid49817.2020.00-80
• 发表时间: 2020-05
• 期刊: 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID)
• 作者: Jashwant Raj Gunasekaran;Michael Cui;P. Thinakaran;Josh Simons;M. Kandemir;C. Das

Cocktail: A Multidimensional Optimization for Model Serving in Cloud

Cocktail：云中模型服务的多维优化

• 发表时间: 2023
• 期刊: 19th USENIX Symposium on Networked Systems Design and Implementation.
• 作者: Jashwant Raj Gunasekaran, Cyan Subhra

Efficient mapping of accurate long reads in minimizer space with mapquik.

• DOI: 10.1101/gr.277679.123
• 发表时间: 2023-07
• 期刊: GENOME RESEARCH
• 影响因子: 7
• 作者: Ekim, Baris;Sahlin, Kristoffer;Medvedev, Paul;Berger, Bonnie;Chikhi, Rayan
• 通讯作者: Chikhi, Rayan