Making Big and Complex Data Easier to Assemble and Analyze in Distributed CI Environments: Expanding on Metagenomics Challenges Defined by CAMERA

用于高级微生物生态学研究和分析的社区网络基础设施（CAMERA，http：//camera.calit2.net/）是语义使能的数据库和分布式计算基础设施，其提供用于存放、定位、分析、可视化和共享微生物生物学数据的单个系统。随着新的DNA测序方法的快速发展，所谓的下一代测序（NGS）技术，如Illumina HiSeq和MiSeq，研究人员使用测序数据来满足现有方法对大规模NGS数据集的计算需求变得越来越困难。为了应对大数据挑战的这些方面，CAMERA团队正在开发新的生物信息学算法、高性能计算解决方案、可视化界面和数据资源，以专门应对NGS数据分析挑战。在这里，该小组提出了一种用于分析NGS数据的横切方法，该方法将创新的生物信息学算法和工作流程与管理大规模分布式计算的前沿计算方法结合在一起。用于大数据分析的XSEDE资源的集成将提供驱动该系统开发所需的规模和规范。该项目将在两年多的时间里进行。第一年将专注于核心CAMERA CI（例如Panfish）的改进和核心NGS工作流程/算法的持续开发。具体来说，CAMERA CI将得到扩展，以充分利用将于2015年初投入使用的两个新的NSF XSEDE资源（SDSC TACC彗星的牧马人）。 第二年将专注于Wrangler和Comet的生产集成，以及随后将NGS工作流程（通过CAMERA CI）部署到整个CAMERA社区。 这些新的软件工具和流水线流程有助于处理和分析每个样品数十GB规模的超大规模宏基因组数据，并提供全面和独特的功能，如16 S分析[7]，分类分箱[8]，组装，rRNA查找，聚类，过滤，功能和途径注释以及可视化。这些下一代工具能够实现数量级更快的计算过程、更全面的分析、集成的数据输出以及调查复杂数据的新方法，一旦这些工具在可扩展的HPC云环境中运行。更广泛的影响被认为是目前，由于过程的复杂性和涉及的软件工具的大量，手动操作是必要的，以完成这些工具的分析。该项目的目标是开发一系列完全集成和易于使用的分析工作流程，封装这些工具。这些新的软件工具工作流程将显著改善使用宏基因组学作为调查工具的研究人员的NGS数据分析，这些研究人员现在受到管理和分析大数据方面的挑战的阻碍。

The Community Cyberinfrastructure for Advanced Microbial Ecology Research and Analysis (CAMERA, http://camera.calit2.net/) is a semantically enabled database and distributed computational infrastructure that provides a single system for depositing, locating, analyzing, visualizing, and sharing microbial biology data. With the rapid advance of newer DNA sequencing methods, so called Next Generation Sequencing (NGS) technologies, such as Illumina HiSeq and MiSeq, it is becoming increasingly difficult for researchers using sequencing data to meet the computing requirements for large-scale NGS datasets with existing methods. In response to these aspects of the BIG DATA challenge, the CAMERA team is developing new bioinformatics algorithms, high performance computing solutions, visualization interfaces, and data resources to specifically address the NGS data analysis challenges. Here, the group proposes a crosscutting methodology for analyzing NGS data that marries innovative bioinformatics algorithms and workflows with leading edge computational methods for managing large scale distributed computing. The integration of XSEDE resources for BIG DATA analysis will provide the scale and specification necessary to drive the development of this system. This project will be conducted over two years. Year one will be focused on the refinement of core CAMERA CI (e.g. Panfish) and the continued development of core NGS workflows/algorithms. Specifically, CAMERA CI will be extended to take full advantage of two new NSF XSEDE resources to be commissioned in early 2015 (Wrangler at TACC & Comet at SDSC). Year 2 will be focused on the production integration of Wrangler and Comet and the subsequent deployment of the NGS workflows (via CAMERA CI) to the entire CAMERA community. These new software tools and pipelined processes facilitate the processing and analyze very large-scale metagenomic data on the scale of tens of GB per sample and provide comprehensive and unique functions such as 16S analysis[7], taxonomy binning[8], assembly, rRNA finding, clustering, filtering, function and pathway annotation, and visualization]. These next generation tools enable orders of magnitude faster computational process, more comprehensive analysis, integrated data output, and novel ways to investigate complex data, once made to operate in extensible HPC cloud environments. The Broader Impact is viewed as that currently, manual operations are necessary to complete analysis with these tools due to the complexity of the process and the large number of software tools involved. The goal of this project is to develop a series of fully integrated and easy-to-use analysis workflows encapsulating these tools. These new workflows of software tools will significantly improve NGS data analysis for researchers who use metagenomics as an investigative tool, researchers who are now impeded by challenges with regard to managing and analyzing BIG DATA.