Snowflock: virtual machine framework for parallel and data intensive computing

数十个基因组的测序，数千种蛋白质结构的确定，以及提供生物体中所有基因信息的各种新颖实验技术，已经将生物学转变为一门信息驱动的科学。生物信息学家正在开发与tb级数据集交互的应用程序，然而他们的用户受到无法购买和维护使用这些应用程序所需设备的限制。本提案介绍了一种利用虚拟机（VM）技术革新数据和计算密集型应用程序（特别是生物信息学工具）提供给科学家和工程师的新范式。在新的范例中，大型数据集存储在服务提供商提供的资源上，并且封装在vm中的应用程序迁移到提供商以接近数据。此外，我们通过分叉VM的多个副本来利用集群计算，然后在不同的物理主机上独立执行，并在完成必要的计算后消失。我们将在Snowflock软件包中实现这种新颖的范例，它将有效的VM复制和调度与强大的数据访问例程相结合，以及专门为生物信息学设计的API和分发。我们相信，通过将并行化的优势与VM技术的优势相结合，Snowflock将为更广泛的科学家和工程师社区提供高性能应用程序。

The sequencing of dozens of genomes, determination of the structure of thousands of proteins and variousnovel experimental techniques that provide information about all genes in an organism have transformedbiology into an information-driven science. Bioinformaticians are developing applications that interact withterabyte-sized datasets, yet their users are constrained by the inability to purchase and maintain the equipmentnecessary to use these applications. This proposal introduces a novel paradigm that leverages virtual machine(VM) technology to revolutionize the way in which data- and compute- intensive applications in general, andbioinformatics tools in particular, are provided to scientists and engineers. In the new paradigm, large datasetsare stored on resource provided by a service provider, and applications encapsulated inside VMs migrate to theprovider to be close to the data. Moreover we take advantage of cluster computing by forking multiple copiesof a VM, which then execute independently on different physical hosts and disappear upon the completion ofthe necessary computation. We will implement this novel paradigm in the Snowflock package, which willcombine efficient VM replication and scheduling with powerful data-access routines, and an API anddistribution specifically designed for bioinformatics. We believe that by combining the benefits ofparallelization with the strengths of VM technology, Snowflock will make high performance applicationsaccessible to a much broader community of scientists and engineers.