SparkRA: Enabling Big Data Scalability for the GATK RNA-seq Pipeline with Apache Spark
SparkRA: Enabling Big Data Scalability for the GATK RNA-seq Pipeline with Apache Spark
复制标题
DOI：
10.3390/genes11010053
复制
发表时间：
2020-01-01
期刊：
影响因子：
3.5
通讯作者：
Mushtaq, Hamid
中科院分区：
文献类型：
--
作者：
Al-Ars, Zaid;Wang, Saiyi;Mushtaq, Hamid
低成本RNA-seq数据的快速增长导致人们对用于各种应用的RNA分析技术越来越感兴趣，从鉴定基因型-表型关系到验证其他分析结果的发现。然而，该领域中的许多实际应用受到可用计算资源和执行分析所需的相关联的长计算时间的限制。GATK有一个流行的最佳实践管道，专为变异调用RNA-seq分析而设计。该管道中的一些工具没有进行优化，无法有效地将分析扩展到多个处理器或计算节点，从而限制了它们处理大型数据集的能力。在本文中，我们提出了SparkRA，这是一种基于Apache Spark的管道，可以在一个节点或大型集群中的多个核心上有效地扩展GATK RNA-seq变体调用管道。在具有20个超线程核心的单个节点上，原始管道运行超过5小时，以处理32 GB的数据集。相比之下，SparkRA能够将同一单个节点上的流水线的总计算时间减少约4倍，将计算时间减少到1.3小时。在具有16个节点（每个节点具有8个单线程核心）的集群上，与单个节点相比，SparkRA能够将计算时间进一步减少7.7倍。与其他可扩展的最先进的解决方案相比，SparkRA速度快1.2倍，同时实现相同的结果精度。
The rapid proliferation of low-cost RNA-seq data has resulted in a growing interest in RNA analysis techniques for various applications, ranging from identifying genotype-phenotype relationships to validating discoveries of other analysis results. However, many practical applications in this field are limited by the available computational resources and associated long computing time needed to perform the analysis. GATK has a popular best practices pipeline specifically designed for variant calling RNA-seq analysis. Some tools in this pipeline are not optimized to scale the analysis to multiple processors or compute nodes efficiently, thereby limiting their ability to process large datasets. In this paper, we present SparkRA, an Apache Spark based pipeline to efficiently scale up the GATK RNA-seq variant calling pipeline on multiple cores in one node or in a large cluster. On a single node with 20 hyper-threaded cores, the original pipeline runs for more than 5 h to process a dataset of 32 GB. In contrast, SparkRA is able to reduce the overall computation time of the pipeline on the same single node by about 4x, reducing the computation time down to 1.3 h. On a cluster with 16 nodes (each with eight single-threaded cores), SparkRA is able to further reduce this computation time by 7.7x compared to a single node. Compared to other scalable state-of-the-art solutions, SparkRA is 1.2x faster while achieving the same accuracy of the results.