HySim: Hybrid-parallel similarity search for the analysis of big genomic and proteomic data

HySim：用于分析大基因组和蛋白质组数据的混合并行相似性搜索

• 批准号: 329350978
• 负责人: Professor Dr. Andreas Hildebrandt
• 金额: --
• 依托单位: Arbeitsgruppe Parallele und Verteilte Architekturen
• 依托单位国家: 德国
• 项目类别: Research Grants
• 财政年份: 2016
• 资助国家: 德国
• 起止时间: 2015-12-31 至 2020-12-31
• 项目状态: 已结题
• 来源: https://gepris.dfg.de/gepris/projekt/329350978?language=en
• 关键词: HySim; Hybrid; parallel; similarity; search

Recent years have seen a tremendous increase in the volume of data generated in the life sciences. The analysis of these data sets poses difficult computational challenges and is an active field of research. Currently, a popular strategy in data rich scenarios across many areas of science and industry is to adopt big data technologies. However, characteristics of typical biological data sets and their intended uses differ significantly from most other big data application areas. Biological data processing often requires more complex analysis techniques than can be afforded by big data technology, which is often constrained to algorithms or heuristics with linear or sublinear complexity. In many application scenarios, rough approximations of true outcomes are perfectly acceptable, but in the life sciences, this is rarely the case. A biomedical application will typically be unable to tolerate even moderate numbers of classification mistakes. Consequently, computational life sciences today tend to rely on a different computational model for large scale applications, namely high performance computing (HPC). However, HPC is tailored more towards problems with a significant amount of computational work (big compute) than at those with enormous storage requirements (big data). The peculiarities of biological data sets and the complexity of the required data analysis pose challenges that neither of the two approaches is perfectly suited to overcome. Instead, a hybrid approach, combining ideas from big data with HPC methodologies, might be preferable, as ideas from big data algorithms can help flexible and highly performant HPC methods to scale towards data sets that would otherwise be too large for them.In this project, we propose to study such hybrid methods in order to meet the challenge of processing large scale genomic and proteomic data sets efficiently yet accurately. Our particular focus is similarity search; an important algorithmic technique for a number of applications in both genomics and proteomics. Corresponding data sets are produced by two types of high throughput technologies: Next Generation Sequencers (NGS) and Mass Spectrometers (MS).Our specific project goals are threefold: (i) Design of efficient and accurate big data algorithms for similarity search in NGS data with applications to metagenomics and read error correction based on locality sensitive hashing (LSH) techniques. (ii) Design of efficient and accurate big data algorithms for similarity search in MS raw data with applications to proteomics based on LSH techniques. (iii) Development of efficient implementations of these new algorithms on a hybrid big data/HPC platform that provide strong scalability for large scale NGS and MS data sets.

近年来，生命科学产生的数据量大幅增加。这些数据集的分析提出了困难的计算挑战，是一个活跃的研究领域。目前，在许多科学和工业领域的数据丰富场景中，一种流行的策略是采用大数据技术。 然而，典型生物数据集的特征及其预期用途与大多数其他大数据应用领域有很大不同。生物数据处理通常需要比大数据技术所能提供的更复杂的分析技术，大数据技术通常限于具有线性或次线性复杂度的算法或算法。在许多应用场景中，真实结果的粗略近似是完全可以接受的，但在生命科学中，情况很少如此。生物医学应用程序通常无法容忍甚至中等数量的分类错误。因此，当今的计算生命科学倾向于依赖于用于大规模应用的不同计算模型，即高性能计算（HPC）。 然而，HPC更多地针对具有大量计算工作（大计算）的问题，而不是那些具有巨大存储需求（大数据）的问题。 生物数据集的特殊性和所需数据分析的复杂性提出了两种方法都不适合克服的挑战。相反，一个混合的方法，结合大数据的思想与HPC方法，可能是可取的，因为大数据算法的想法可以帮助灵活和高性能的HPC方法，以扩展到数据集，否则将是太大了他们。在这个项目中，我们建议研究这种混合方法，以满足大规模基因组和蛋白质组数据集的处理效率和准确性的挑战。我们特别关注的是相似性搜索，在基因组学和蛋白质组学的一些应用程序的重要算法技术。相应的数据集由两种高通量技术产生：下一代测序仪（NGS）和质谱仪（MS）。我们的具体项目目标有三个方面：（i）设计高效准确的大数据算法，用于NGS数据中的相似性搜索，并应用于宏基因组学和基于局部敏感哈希（LSH）技术的读取错误校正。(ii)设计高效准确的大数据算法，用于MS原始数据中的相似性搜索，并应用于基于LSH技术的蛋白质组学。(iii)在混合大数据/HPC平台上开发这些新算法的有效实现，为大规模NGS和MS数据集提供强大的可扩展性。

项目成果

Gossip: Efficient Communication Primitives for Multi-GPU Systems

• DOI: 10.1145/3337821.3337889
• 发表时间: 2019-08
• 期刊: Proceedings of the 48th International Conference on Parallel Processing
• 作者: Robin Kobus;Daniel Jünger;Christian Hundt;B. Schmidt

WarpCore: A Library for fast Hash Tables on GPUs

WarpCore：GPU 上的快速哈希表库

• DOI: 10.1109/hipc50609.2020.00015
• 发表时间: 2020
• 期刊: 2020 IEEE 27th International Conference on High Performance Computing, Data, and Analytics (HiPC)
• 作者: D. Jünger;R. Kobus;A. Müller;C. Hundt;W. Liu;B. Schmidt
• 通讯作者: B. Schmidt

Locality-sensitive hashing enables signal classification in high-throughput mass spectrometry raw data at scale

• DOI: 10.1101/2021.07.01.450702
• 发表时间: 2021-07
• 期刊: bioRxiv
• 作者: Konstantin Bob;David Teschner;T. Kemmer;David Gomez-Zepeda;S. Tenzer;B. Schmidt;A. Hildebrandt

CARE: context-aware sequencing read error correction

• DOI: 10.1093/bioinformatics/btaa738
• 发表时间: 2021-04-01
• 期刊: BIOINFORMATICS
• 影响因子: 5.8
• 作者: Kallenborn, Felix;Hildebrandt, Andreas;Schmidt, Bertil
• 通讯作者: Schmidt, Bertil

RNACache: Fast Mapping of RNA-Seq Reads to Transcriptomes Using MinHashing

RNACache：使用 MinHashing 将 RNA-Seq 读取快速映射到转录组

• DOI: 10.1007/978-3-030-77961-0_31
• 发表时间: 2021
• 作者: J. Cascitti;S. Niebler;A. Müller;B. Schmidt
• 通讯作者: B. Schmidt