Kmerind: A Flexible Parallel Library for K-mer Indexing of Biological Sequences on Distributed Memory Systems
Kmerind: A Flexible Parallel Library for K-mer Indexing of Biological Sequences on Distributed Memory Systems
复制标题
DOI：
10.1109/tcbb.2017.2760829
复制
发表时间：
2019-07-01
影响因子：
4.5
通讯作者：
Aluru，Srinivas
中科院分区：
文献类型：
--
作者：
Pan，Tony;Flick，Patrick;Aluru，Srinivas
计数和索引生物序列中的固定长度子串或k-mer是许多生物信息学任务中的关键步骤，包括基因组比对和映射、基因组组装和纠错。虽然下一代测序技术的进步已经大大降低了成本并改善了延迟和通量，但几乎没有生物信息学工具和文库可以以每3天1.8个碱基的当前生成速率有效地处理数据集。我们提出Kmerind，一个高性能的k-mer索引库的分布式内存环境。Kmerind库提供了一组简单且一致的API，具有顺序语义和并行实现，设计灵活且可扩展。使用Kmerind，用户可以很容易地从内置或用户提供的组件实例化应用程序特定的索引，如k-mer计数器和位置索引，而无需广泛的高性能计算专业知识。Kmerind的k-mer计数器的性能类似于或优于现有的同类最佳k-mer计数工具，即使在共享内存系统上。在分布式内存环境中，Kmerind在1024个Xeon CPU内核上，在不到13秒的时间内对120 GB序列读取数据集中的k-mer进行计数，并在大约17秒内完全索引它们的位置。查询这些索引中1%的k-mer可以分别在0.23秒和28秒内完成。据我们所知，Kmerind是第一个用于分布式内存环境的k-mer索引库，也是第一个完全可定制和可扩展的通用k-mer索引和计数库。Kmerind可从https://github.com/ParBLiSS/kmerind获得。
Counting and indexing fixed length substrings, or k-mers, in biological sequences is a key step in many bioinformatics tasks including genome alignment and mapping, genome assembly, and error correction. While advances in next generation sequencing technologies have dramatically reduced the cost and improved latency and throughput, there exist few bioinformatics tools and libraries that can efficiently process the data sets at the current generation rate of 1.8 terabases every 3 days. We present Kmerind, a high performance k-mer indexing library for distributed memory environments. The Kmerind library provides a set of simple and consistent APIs with sequential semantics and parallel implementations that are designed to be flexible and extensible. Using Kmerind, a user can easily instantiate application-specific indices, such as k-mer counter and position index, from biult-in or user-supplied components without extensive high performance computing expertise. Kmerind's k-mer counter performs similarly or better than existing, best-in-class k-mer counting tools even on shared memory systems. In a distributed memory environment, Kmerind counts k-mers in a 120 GB sequence read data set in less than 13 seconds on 1024 Xeon CPU cores, and fully indexes their positions in approximately 17 seconds. Querying for 1% of the k-mers in these indices can be completed in 0.23 seconds and 28 seconds, respectively. To our knowledge, Kmerind is the first k-mer indexing library for distributed memory environments, and the first fully customizable and extensible library for general k-mer indexing and counting. Kmerind is available from https://github.com/ParBLiSS/kmerind.