CIF: Small: Distributed Storage Systems from Combinatorial Designs

大型内容共享、社交网站和云存储提供商需要以分布式方式存储大量数据（以数百pb为数量级）。在这些系统中，节点故障和数据丢失是常态，而不是例外。因此，研究存储技术和算法(a)允许在节点故障的情况下恢复数据和(b)允许故障节点的有效再生是至关重要的。当然，最好以分布式方式执行再生，并优化与之相关的性能指标。例如，希望再生过程快速，并且只接触几个节点进行修复。研究人员将研究存储方法，允许非常简单和有效的修复，同时允许在存在大量故障的情况下恢复数据。该研究涉及对分布式存储系统的调查，该系统具有对多次故障的弹性，并且可以以精确和非编码的方式进行修复；也就是说，新节点可以通过简单地从幸存节点下载数据包来生成故障节点的精确副本，而不需要计算开销。复制代码越来越多地部署在大型互联网公司使用的大型数据中心中。重建代码方面的进步预计将为这些数据中心节省数百万美元的维护成本。为了保证本研究结果的可访问性，将使用开源的Hadoop分布式文件系统进行实施，Hadoop是分布式云存储前沿研究和开发的首选平台。

Large scale content sharing, social networking sites and cloud storage providers require massive amounts of data (on the order of hundreds of petabytes) to be stored in a distributed manner. In these systems, node failures and loss of data are the norm rather than the exception. Thus, it is of critical importance to investigate storage techniques and algorithms that (a) allow for data recovery in the presence of node failures and (b) allow for efficient regeneration of failed nodes. It is, of course, desirable to perform the regeneration in a distributed manner and optimize performance metrics associated with it. For instance, it is desirable that the regeneration process be fast and that only a few nodes be contacted for the repair. The investigator will research storage methods that allow for very simple and efficient repair while simultaneously allowing for data recovery in the presence of a large number of failures. The research involves the investigation of distributed storage systems that are resilient to multiple failures and can be repaired in an exact and uncoded manner; i.e., the new node may produce an exact copy of the failed node by simply downloading packets from the surviving nodes, with no computational overhead. Replication codes are increasingly deployed in large-scale data centers in use by major internet companies. Advances in reconstruction codes are expected to translate into millions of dollars in savings in maintenance costs in such data centers. To ensure accessibility of the results of this research, implementation will be carried out using the open source Hadoop distributed file system, the preferred platform for cutting-edge research and development in distributed cloud storage.