Continued Development and Maintenance of the MG-RAST Metagenomics Pipeline

 描述(申请人提供)：元基因组学是对直接从环境中采样的微生物种群的研究，为通过微生物图谱发现新的酶；使用微生物变化作为健康预测指标；或衡量矿产开采等人类活动的可持续性提供了途径。然而，元基因组数据量很大(例如，人类肠道微生物区系的元基因组的大小约为1千兆字节)，需要进行的处理才能从大型数据集中提取意义，例如通过与不断更新的知识数据库进行比较来识别样本中的生物基因组(分类注释)以及它们在做什么(功能注释)。随着实验者要求进行越来越多的元基因组分析，这些数字只会不断增长。由于这种需求，我们的MG-RAST(元基因组学-快速注释)门户网站是一项开源、高通量的元基因组学服务，自2008年以来一直是主要的社区资源，容纳了超过160K个数据集和40K用户。然而，自最初的设计以来，MG-RAST见证了下一代测序技术的疯狂发展，极大地改变了计算环境(硬件和软件)，改变了对用户数和数据集的数量和多样性的要求，增加了流水线组件的复杂性，以及对更高吞吐量的要求。为了适应这种情况，MG-RAST一直在不断地进行修改。修改包括通过几项算法改进升级管道组件；部署定制的数据和工作流管理系统-Shock对象存储和AWE工作流管理器；以及将MG-RAST移植到基于云的分布式体系结构。尽管我们不断地改进特别的系统，但我们的初步研究表明，有必要全面重新设计MG-RAST，以跟上快速发展的元基因组学领域的需求。我们提出的增强是基于表达的用户需求、新的使用模式以及结合新工具的灵活性，特别是对于查询序列的计算密集型相似性分析。通过这个项目，我们建议通过以下方式完成MG-RAST的转型：(I)改进其功能和数据再现性；(Ii)通过自动监控和生成测试套件来提高其软件质量和性能；以及(Iii)迈向元基因组数据的联合基础设施。总体而言，我们目标的成功实现将通过服务和数据的联合支持替代的元基因组学服务模式，并产生强大的最先进的元基因组学资源。生物医学管道的联合总体上是一个强有力的方向，可以利用不同用户群的专业知识，并反过来使其用户受益。因此，MG-RAST作为一条最先进的管道，将能够支持不断增长的用户基础，处理更大和更多样化的数据集，并与新的基因组学技术一起进化。这最终的目标是加速终端用户应用程序的进步，例如根据患者的微生物群定制的个性化药物。

 DESCRIPTION (provided by applicant): Metagenomics, the study of microbial populations sampled directly from the environment, affords avenues for discovering novel enzymes via microbial profiling; using microbial shifts as predictors for health; or gauging the sustainabilityof human operations like mineral mining. However, the volume of metagenomic data is large (e.g., the metagenome of a human's gut microbiota is about 1 Gigabasepairs in size) and the processing that needs to be done to extract meaning out of the large datasets is significant, such as to identify what organisms' genomes are in the sample (taxonomic annotation) and what are they doing (functional annotation) via comparisons with continually updated knowledge databases. These numbers are only growing as experimentalists demand more and more metagenomic analysis runs. Borne out of this need, our MG-RAST (Metagenomics-Rapid Annotation) portal, an open-source, high-throughput, metagenomics service, has been a major community resource since 2008, housing over 160K datasets and 40K users. However, since its original design, MG-RAST has witnessed the frenetic development of next-generation sequencing technologies, drastically altered computing landscape (both in hardware and software), changed requirements in terms of number of users and datasets' volumes and diversity, increasing complexity of pipeline components, and requirements for higher throughput. To adapt to this, MG-RAST has been continually modified. Modifications included upgrading the pipeline components with several algorithmic improvements; deploying a customized data and workflow management system - the SHOCK object store and AWE workflow manager; and porting MG-RAST to a cloud-based distributed architecture. Notwithstanding our continual, albeit ad-hoc system improvements, our pilot studies have indicated the need for a comprehensive redesign of MG-RAST to keep pace with the needs of the rapidly advancing field of metagenomics. Our proposed enhancements are based on expressed user requirements, new usage patterns, and flexibility to incorporate new tools, especially for the compute-intensive similarity analysis for queried sequences. Through this project, we propose to accomplish MG-RAST's transformation via (i) improving its functionality and data reproducibility; (ii) improving its software quality and performance through automated monitoring and generation of test suites; and (iii) moving toward a federated infrastructure for metagenomics data. Overall, the successful accomplishment of our aims will support alternate metagenomics service models through federation of services and data and result in a robust state-of-the-art metagenomics resource. Federation in biomedical pipelines is in general a powerful direction to leverage the expertise of diverse user-bases and, reciprocally, benefit its users. Thus, MG-RAST, as a state- of-the-art pipeline, will be capable of supporting an ever increasing user-base, handling larger and more varied datasets, and evolving in concert with new genomics technologies. This, with the ultimate goal, to accelerate advances in end-user applications, e.g., personalized medicine, tailored to the patient's microbiome.