Fast crash recovery strategies for many small data objects in a distributed memory storageAkronym: FastRecovery

越来越多的程序需要管理数十亿个小数据对象，例如社交网络应用程序。磁盘和固态驱动器的数据访问时间对于这些交互式应用程序来说太慢，并且供应商被迫将许多数据保存在缓存中。对于大型应用程序，数据无法加载到单个节点的内存中，因此需要聚合潜在许多节点的内存。一个突出的例子是Facebook运行超过1,000个memcached服务器，将大约75%的数据始终保存在内存中，因为后台数据库太慢了。显然，在节点故障和断电的情况下，数据会丢失，并且可能需要数小时才能从数据库或文件系统等辅助存储加载大量数据。拟议的项目通过开发和评估分布式内存系统的快速恢复策略来解决这些挑战。该项目专注于一个键值数据模型，用于多达1万亿个小数据对象（大小约为16-64字节，存储在1,000个节点中）。恢复将使用根据日志结构文件系统的研究针对SSD驱动器优化的异步日志记录策略。一个节点的状态需要分布在多个备份节点上，以便实现快速和并行的恢复。属于一个节点状态的所有日志部分也需要被复制，以便能够屏蔽永久的节点故障。重要的是要指出，对于大型集群，如果多个节点同时发生故障，随机副本放置有很高的数据丢失概率。我们计划解决这一挑战的基础上，最近提出的Copyset副本放置方案，我们计划制定有效的和自适应的策略，最大限度地减少数据丢失的概率，同时允许快速恢复。备份管理将使用超级对等覆盖网络实施，同时考虑到不同的指标，包括负载和持续恢复以及重新复制。

More and more programs need to manage billions of small data objects like for example social network applications. Data access times of disks and solid state drives are too slow for these interactive applications and providers are forced to keep many data in caches. For large applications data cannot be loaded into memory of a single node and thus the memory of potentially many nodes need to be aggregated. A prominent example is Facebook running more than 1,000 memcached servers to keep around 75% of all data always in memory because background databases are too slow. Obviously, data is lost in case of node failures and power outages and it can take hours to load large data volumes from secondary storage like databases or file systems. The proposed project addresses these challenges by developing and evaluating fast recovery strategies for distributed memory systems. This project focuses on a key-value data-model for up to one trillion small data objects (sizes around 16-64 byte, stored in 1,000 nodes). Recovery will use an asynchronous logging strategy optimized for SSD drives, based on research from log-structured file systems. The state of one node needs to be distributed on many backup nodes in order to allow a fast and parallel recovery. All the log parts belonging to one node state need also to be replicated in order to be able to mask permanent node failures. It is important to point out that random replica placement has a high probability of data loss for large clusters, if several nodes fail simultaneously. We plan to address this challenge based upon the recently proposed Copyset replica placement scheme and we plan to develop efficient and adaptive strategies which minimize data loss probability while at the same time allow fast recovery. The backup management will be implemented using a super-peer overlay network taking into account different metrics including load and ongoing recoveries as well as re-replication.