Collaborative Research: ABI Development: A New Platform for highly-optimized, low-latency pipelines for genomic data analysis

下一代测序已经将基因组学转变为数据密集型计算的新范式，提出了几个突出的挑战。首先，海量的基因组数据需要进行深度分析，挖掘生物信息，这需要一个完整的管道，集成了许多数据处理和分析工具。其次，深度分析管道通常需要很长时间才能运行，这需要很长的算法和方法开发周期。该项目旨在将最新的大数据技术和数据库技术引入基因组学领域，以彻底改变其数据处理能力。预计该项目将产生重大的科学和教育效益。通过为基因组数据分析提供高度优化的并行处理平台，并使其可在私有云和公共云中访问，它将使许多新的模型和算法能够为基因组学开发，并帮助以前所未有的速度推进这一领域，就像大数据技术为互联网公司所做的那样。该项目还将研究和教育与课程开发相结合，为K-12教师和社区大学教师提供辅导，并通过大学推广和NSF资助的推广计划吸引女性参与研究。拟议的研究包括：（1）通过组装最先进的方法开发基因组数据分析的深层管道，（2）使用大数据技术自动并行化工作流程，（3）优化基因组流水线的原则性方法，以及（4）整合流技术以减少重要结果的延迟。该原型系统将部署在私有云和公共云环境中，并使用纽约基因组中心现有的长期运行管道和各种真实的用例进行全面评估。通过这样做，该项目将提供有关如何适应和推进大数据技术的新知识，包括新的优化，分区和调度技术，用于基因组学领域。该项目的结果在网站上公布：http://gesall.cs.umass.edu。

Next-generation sequencing has transformed genomics into a new paradigm of data-intensive computing, raising several salient challenges. First, the deluge of genomic data needs to undergo deep analysis to mine biological information, which requires a full pipeline that integrates many data processing and analysis tools. Second, deep analysis pipelines often take long to run, which entails a long cycle for algorithm and method development. This project aims to bring the latest big data technology and database technology to the genomics domain to revolutionize its data crunching power. This project is anticipated to produce significant scientific and educational benefits. By providing a highly-optimized parallel processing platform for genomic data analysis and making it accessible in private and public clouds, it will enable many new models and algorithms to be developed for genomics and help advance this field at unprecedented speed as big data technology did for Internet companies. This project also integrates research and education with curriculum development, tutorials for K-12 teachers and community college faculty, and engaging women in research through college outreach and NSF-funded outreach programs.The proposed research includes the development of (1) a deep pipeline for genomic data analysis by assembling state-of-the-art methods, (2) automatic parallelization of the workflow using the big data technology, (3) a principled approach to optimizing the genomic pipeline, and (4) integration of streaming technology to reduce latency of important results. The prototype system will be deployed in both private and public cloud environments, and fully evaluated using existing long-running pipelines at the New York Genome Center and in a variety of real use cases. By way of doing so, this project will provide new knowledge regarding how to adapt and advance big data technology, including new optimization, partitioning, and scheduling techniques, for the genomics domain. The results of the project are disseminated at the web site: http://gesall.cs.umass.edu.