Collaborative Research: ABI Development: A New Platform for highly-optimized, low-latency pipelines for genomic data analysis

下一代测序将基因组学转变为数据密集型计算的新范式，提出了几个突出的挑战。首先，海量的基因组数据需要进行深度分析来挖掘生物信息，这需要一个完整的管道，集成了许多数据处理和分析工具。其次，深度分析管道通常需要很长时间才能运行，这需要很长的算法和方法开发周期。该项目旨在将最新的大数据技术和数据库技术引入基因组学领域，以彻底改变其数据处理能力。该项目预计将产生重大的科学和教育效益。通过为基因组数据分析提供高度优化的并行处理平台，并使其可在私有云和公共云中访问，它将使许多新的基因组学模型和算法得以开发，并帮助该领域以前所未有的速度向前发展，就像大数据技术对互联网公司所做的那样。该项目还将研究和教育与课程开发、K-12教师和社区大学教师的教程、以及通过大学外展和美国国家科学基金会资助的外展项目吸引女性参与研究相结合。建议的研究包括：(1)通过整合最先进的方法开发基因组数据分析的深层管道，(2)使用大数据技术实现工作流的自动并行化，(3)优化基因组管道的原则方法，以及(4)集成流技术以减少重要结果的延迟。原型系统将部署在私有云和公共云环境中，并使用纽约基因组中心现有的长期运行管道和各种实际用例进行全面评估。通过这种方式，该项目将为基因组学领域提供关于如何适应和推进大数据技术的新知识，包括新的优化、分区和调度技术。该项目的结果已在网站http://gesall.cs.umass.edu上公布。

Next-generation sequencing has transformed genomics into a new paradigm of data-intensive computing, raising several salient challenges. First, the deluge of genomic data needs to undergo deep analysis to mine biological information, which requires a full pipeline that integrates many data processing and analysis tools. Second, deep analysis pipelines often take long to run, which entails a long cycle for algorithm and method development. This project aims to bring the latest big data technology and database technology to the genomics domain to revolutionize its data crunching power. This project is anticipated to produce significant scientific and educational benefits. By providing a highly-optimized parallel processing platform for genomic data analysis and making it accessible in private and public clouds, it will enable many new models and algorithms to be developed for genomics and help advance this field at unprecedented speed as big data technology did for Internet companies. This project also integrates research and education with curriculum development, tutorials for K-12 teachers and community college faculty, and engaging women in research through college outreach and NSF-funded outreach programs.The proposed research includes the development of (1) a deep pipeline for genomic data analysis by assembling state-of-the-art methods, (2) automatic parallelization of the workflow using the big data technology, (3) a principled approach to optimizing the genomic pipeline, and (4) integration of streaming technology to reduce latency of important results. The prototype system will be deployed in both private and public cloud environments, and fully evaluated using existing long-running pipelines at the New York Genome Center and in a variety of real use cases. By way of doing so, this project will provide new knowledge regarding how to adapt and advance big data technology, including new optimization, partitioning, and scheduling techniques, for the genomics domain. The results of the project are disseminated at the web site: http://gesall.cs.umass.edu.