Fast crash recovery strategies for many small data objects in a distributed memory storageAkronym: FastRecovery

越来越多的程序需要管理数十亿个小数据对象，比如社交网络应用。对于这些交互式应用程序来说，磁盘和固态硬盘的数据访问时间太慢，供应商被迫将许多数据保存在缓存中。对于大型应用程序，不能将数据加载到单个节点的内存中，因此需要聚合潜在多个节点的内存。一个突出的例子是Facebook运行超过1,000台Memcached服务器，将大约75%的数据始终保存在内存中，因为后台数据库太慢了。显然，在节点故障和停电的情况下，数据会丢失，从辅助存储(如数据库或文件系统)加载大量数据卷可能需要几个小时。拟议的项目通过开发和评估分布式存储系统的快速恢复策略来应对这些挑战。该项目的重点是为多达1万亿个小数据对象(大小约为16-字节，存储在1,000个节点中)的键值数据模型。根据对日志结构文件系统的研究，Recovery将使用针对SSD驱动器进行优化的异步日志记录策略。一个节点的状态需要分布在多个备份节点上，才能实现快速并行恢复。还需要复制属于一个节点状态的所有日志部分，以便能够屏蔽永久节点故障。需要指出的是，如果多个节点同时发生故障，则随机副本放置对于大型集群而言具有很高的数据丢失概率。我们计划在最近提出的Copyset副本放置方案的基础上解决这一挑战，并计划开发高效和自适应的策略，使数据丢失概率最小化，同时允许快速恢复。备份管理将使用超级对等覆盖网络实施，同时考虑到不同的指标，包括负载和持续恢复以及重新复制。

More and more programs need to manage billions of small data objects like for example social network applications. Data access times of disks and solid state drives are too slow for these interactive applications and providers are forced to keep many data in caches. For large applications data cannot be loaded into memory of a single node and thus the memory of potentially many nodes need to be aggregated. A prominent example is Facebook running more than 1,000 memcached servers to keep around 75% of all data always in memory because background databases are too slow. Obviously, data is lost in case of node failures and power outages and it can take hours to load large data volumes from secondary storage like databases or file systems. The proposed project addresses these challenges by developing and evaluating fast recovery strategies for distributed memory systems. This project focuses on a key-value data-model for up to one trillion small data objects (sizes around 16-64 byte, stored in 1,000 nodes). Recovery will use an asynchronous logging strategy optimized for SSD drives, based on research from log-structured file systems. The state of one node needs to be distributed on many backup nodes in order to allow a fast and parallel recovery. All the log parts belonging to one node state need also to be replicated in order to be able to mask permanent node failures. It is important to point out that random replica placement has a high probability of data loss for large clusters, if several nodes fail simultaneously. We plan to address this challenge based upon the recently proposed Copyset replica placement scheme and we plan to develop efficient and adaptive strategies which minimize data loss probability while at the same time allow fast recovery. The backup management will be implemented using a super-peer overlay network taking into account different metrics including load and ongoing recoveries as well as re-replication.