Making Big and Complex Data Easier to Assemble and Analyze in Distributed CI Environments: Expanding on Metagenomics Challenges Defined by CAMERA

使大而复杂的数据在分布式 CI 环境中更容易组装和分析：扩展 CAMERA 定义的宏基因组挑战

• 批准号: 1419196
• 负责人: Mark Ellisman
• 金额: $ 25万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-02-15 至 2016-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1419196&HistoricalAwards=false
• 关键词: Making; Big; Complex; Data; Easier

The Community Cyberinfrastructure for Advanced Microbial Ecology Research and Analysis (CAMERA, http://camera.calit2.net/) is a semantically enabled database and distributed computational infrastructure that provides a single system for depositing, locating, analyzing, visualizing, and sharing microbial biology data. With the rapid advance of newer DNA sequencing methods, so called Next Generation Sequencing (NGS) technologies, such as Illumina HiSeq and MiSeq, it is becoming increasingly difficult for researchers using sequencing data to meet the computing requirements for large-scale NGS datasets with existing methods. In response to these aspects of the BIG DATA challenge, the CAMERA team is developing new bioinformatics algorithms, high performance computing solutions, visualization interfaces, and data resources to specifically address the NGS data analysis challenges. Here, the group proposes a crosscutting methodology for analyzing NGS data that marries innovative bioinformatics algorithms and workflows with leading edge computational methods for managing large scale distributed computing. The integration of XSEDE resources for BIG DATA analysis will provide the scale and specification necessary to drive the development of this system. This project will be conducted over two years. Year one will be focused on the refinement of core CAMERA CI (e.g. Panfish) and the continued development of core NGS workflows/algorithms. Specifically, CAMERA CI will be extended to take full advantage of two new NSF XSEDE resources to be commissioned in early 2015 (Wrangler at TACC & Comet at SDSC). Year 2 will be focused on the production integration of Wrangler and Comet and the subsequent deployment of the NGS workflows (via CAMERA CI) to the entire CAMERA community. These new software tools and pipelined processes facilitate the processing and analyze very large-scale metagenomic data on the scale of tens of GB per sample and provide comprehensive and unique functions such as 16S analysis[7], taxonomy binning[8], assembly, rRNA finding, clustering, filtering, function and pathway annotation, and visualization]. These next generation tools enable orders of magnitude faster computational process, more comprehensive analysis, integrated data output, and novel ways to investigate complex data, once made to operate in extensible HPC cloud environments. The Broader Impact is viewed as that currently, manual operations are necessary to complete analysis with these tools due to the complexity of the process and the large number of software tools involved. The goal of this project is to develop a series of fully integrated and easy-to-use analysis workflows encapsulating these tools. These new workflows of software tools will significantly improve NGS data analysis for researchers who use metagenomics as an investigative tool, researchers who are now impeded by challenges with regard to managing and analyzing BIG DATA.

高级微生物生态学研究和分析社区网络基础设施（CAMERA，http://camera.calit2.net/）是一个语义数据库和分布式计算基础设施，提供用于存储、定位、分析、可视化和共享微生物生物学数据的单一系统。随着新型DNA测序方法，即所谓的下一代测序（NGS）技术（例如Illumina HiSeq和MiSeq）的快速发展，研究人员使用测序数据通过现有方法满足大规模NGS数据集的计算要求变得越来越困难。为了应对大数据挑战的这些方面，CAMERA 团队正在开发新的生物信息学算法、高性能计算解决方案、可视化界面和数据资源，以专门解决 NGS 数据分析挑战。在此，该小组提出了一种用于分析 NGS 数据的横切方法，该方法将创新的生物信息学算法和工作流程与用于管理大规模分布式计算的前沿计算方法相结合。用于大数据分析的 XSEDE 资源的集成将提供推动该系统开发所需的规模和规范。该项目将持续两年。第一年将重点关注核心 CAMERA CI（例如 Panfish）的完善以及核心 NGS 工作流程/算法的持续开发。具体来说，CAMERA CI 将进行扩展，以充分利用将于 2015 年初投入使用的两个新的 NSF XSEDE 资源（TACC 的 Wrangler 和 SDSC 的 Comet）。 第 2 年将重点关注 Wrangler 和 Comet 的生产集成，以及随后将 NGS 工作流程（通过 CAMERA CI）部署到整个 CAMERA 社区。 这些新的软件工具和流水线流程有助于处理和分析每个样本数十GB规模的超大规模宏基因组数据，并提供全面且独特的功能，例如16S分析[7]、分类分箱[8]、组装、rRNA查找、聚类、过滤、功能和通路注释以及可视化]。这些下一代工具一旦在可扩展的 HPC 云环境中运行，就能实现计算过程速度加快几个数量级、更全面的分析、集成的数据输出以及研究复杂数据的新颖方法。更广泛的影响是，由于过程复杂且涉及大量软件工具，目前需要手动操作才能使用这些工具完成分析。该项目的目标是开发一系列完全集成且易于使用的封装这些工具的分析工作流程。这些新的软件工具工作流程将显着改善使用宏基因组学作为研究工具的研究人员的 NGS 数据分析，以及现在因管理和分析大数据方面的挑战而受到阻碍的研究人员。