Exploiting Data Reduction Principles in Cloud-Based Data Management for Cryo-Image Data
Exploiting Data Reduction Principles in Cloud-Based Data Management for Cryo-Image Data
复制标题
利用基于云的冷冻图像数据数据管理中的数据缩减原则
DOI：
10.1145/3232174.3232177
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
S. Shakeel
中科院分区：
文献类型：
--
作者：
K. A. Shakil;A. Ora;Mansaf Alam;S. Shakeel
云计算是初创生命科学实验室存储和管理数据的一种经济有效的方式。然而，在许多情况下，存储在云上的数据可能是冗余的，这使得基于云的数据管理效率低下且成本高昂，因为必须为存储在云上的每个字节的数据付费。在这里，我们在基于云的环境中测试了电子冷冻显微镜（cryoEM）实验室生成的数据的有效管理。测试数据来自低温储存库EMPIAR。所有的图像都受到内部并行版本的主成分分析。采用高效的基于云的MapReduce模式进行并行化。我们展示了以tb数量级的大数据可以以经济有效的可扩展方式有效地减少到最小的基本自我。此外，Amazon EC2上的现场实例被证明可以将成本降低约27%。这种方法可以扩展到任何大容量和类型的数据。
Cloud computing is a cost-effective way for start-up life sciences laboratories to store and manage their data. However, in many instances the data stored over the cloud could be redundant which makes cloud-based data management inefficient and costly because one has to pay for every byte of data stored over the cloud. Here, we tested efficient management of data generated by an electron cryo-microscopy (cryoEM) lab on a cloud-based environment. The test data was obtained from cryoEM repository EMPIAR. All the images were subjected to an in-house parallelized version of principal component analysis. An efficient cloud-based MapReduce modality was used for parallelization. We showed that large data in order of terabytes could be efficiently reduced to its minimal essential self in a cost-effective scalable manner. Furthermore, on-spot instance on Amazon EC2 was shown to reduce costs by a margin of about 27 percent. This approach could be scaled to data of any large volume and type.