CIF: Small: Distributed Storage Systems from Combinatorial Designs

大规模内容共享、社交网站和云存储提供商需要以分布式方式存储海量数据(数百PB)。在这些系统中，节点故障和数据丢失是常态，而不是例外。因此，研究以下存储技术和算法至关重要：(A)允许在出现节点故障的情况下进行数据恢复；(B)允许高效地重新生成故障节点。当然，希望以分布式方式执行重新生成并优化与其相关联的性能度量。例如，希望再生过程很快，并且只联系几个节点进行修复。研究人员将研究存储方法，这些方法允许非常简单和高效的修复，同时允许在出现大量故障的情况下进行数据恢复。该研究涉及对分布式存储系统的研究，这些分布式存储系统对多个故障具有弹性，并且可以以精确和非编码的方式进行修复；即，新节点可以通过简单地从幸存节点下载分组来产生故障节点的精确副本，而不需要计算开销。复制代码越来越多地部署在大型互联网公司使用的大型数据中心。重建代码的进步预计将转化为此类数据中心节省数百万美元的维护成本。为了确保这项研究成果的可获得性，将使用开源的Hadoop分布式文件系统进行实施，Hadoop分布式文件系统是分布式云存储前沿研发的首选平台。

Large scale content sharing, social networking sites and cloud storage providers require massive amounts of data (on the order of hundreds of petabytes) to be stored in a distributed manner. In these systems, node failures and loss of data are the norm rather than the exception. Thus, it is of critical importance to investigate storage techniques and algorithms that (a) allow for data recovery in the presence of node failures and (b) allow for efficient regeneration of failed nodes. It is, of course, desirable to perform the regeneration in a distributed manner and optimize performance metrics associated with it. For instance, it is desirable that the regeneration process be fast and that only a few nodes be contacted for the repair. The investigator will research storage methods that allow for very simple and efficient repair while simultaneously allowing for data recovery in the presence of a large number of failures. The research involves the investigation of distributed storage systems that are resilient to multiple failures and can be repaired in an exact and uncoded manner; i.e., the new node may produce an exact copy of the failed node by simply downloading packets from the surviving nodes, with no computational overhead. Replication codes are increasingly deployed in large-scale data centers in use by major internet companies. Advances in reconstruction codes are expected to translate into millions of dollars in savings in maintenance costs in such data centers. To ensure accessibility of the results of this research, implementation will be carried out using the open source Hadoop distributed file system, the preferred platform for cutting-edge research and development in distributed cloud storage.