Fast crash recovery strategies for many small data objects in a distributed memory storageAkronym: FastRecovery

分布式内存存储中许多小数据对象的快速崩溃恢复策略缩写：FastRecovery

• 批准号: 269648469
• 负责人: Professor Dr. Michael Schöttner
• 金额: --
• 依托单位: Fach Informatik
• 依托单位国家: 德国
• 项目类别: Research Grants
• 财政年份: 2015
• 资助国家: 德国
• 起止时间: 2014-12-31 至 2017-12-31
• 项目状态: 已结题
• 来源: https://gepris.dfg.de/gepris/projekt/269648469?language=en
• 关键词: Fast; crash; recovery; strategies; many

More and more programs need to manage billions of small data objects like for example social network applications. Data access times of disks and solid state drives are too slow for these interactive applications and providers are forced to keep many data in caches. For large applications data cannot be loaded into memory of a single node and thus the memory of potentially many nodes need to be aggregated. A prominent example is Facebook running more than 1,000 memcached servers to keep around 75% of all data always in memory because background databases are too slow. Obviously, data is lost in case of node failures and power outages and it can take hours to load large data volumes from secondary storage like databases or file systems. The proposed project addresses these challenges by developing and evaluating fast recovery strategies for distributed memory systems. This project focuses on a key-value data-model for up to one trillion small data objects (sizes around 16-64 byte, stored in 1,000 nodes). Recovery will use an asynchronous logging strategy optimized for SSD drives, based on research from log-structured file systems. The state of one node needs to be distributed on many backup nodes in order to allow a fast and parallel recovery. All the log parts belonging to one node state need also to be replicated in order to be able to mask permanent node failures. It is important to point out that random replica placement has a high probability of data loss for large clusters, if several nodes fail simultaneously. We plan to address this challenge based upon the recently proposed Copyset replica placement scheme and we plan to develop efficient and adaptive strategies which minimize data loss probability while at the same time allow fast recovery. The backup management will be implemented using a super-peer overlay network taking into account different metrics including load and ongoing recoveries as well as re-replication.

越来越多的程序需要管理数十亿的小数据对象，例如社交网络应用程序。对于这些交互式应用程序来说，磁盘和固态驱动器的数据访问时间太慢，提供商被迫将许多数据保存在缓存中。对于大型应用程序，数据不能加载到单个节点的内存中，因此可能需要聚合许多节点的内存。一个突出的例子是Facebook运行了1000多台memcached服务器，因为后台数据库太慢了，所以大约75%的数据总是保存在内存中。显然，在节点故障和断电的情况下，数据会丢失，并且可能需要数小时才能从数据库或文件系统等辅助存储加载大量数据。提议的项目通过开发和评估分布式内存系统的快速恢复策略来解决这些挑战。这个项目关注的是一个键值数据模型，它适用于多达一万亿的小型数据对象（大小约为16-64字节，存储在1,000个节点中）。根据对日志结构文件系统的研究，恢复将使用针对SSD驱动器优化的异步日志记录策略。一个节点的状态需要分布在许多备份节点上，以便允许快速并行恢复。还需要复制属于一个节点状态的所有日志部分，以便能够掩盖永久性节点故障。需要指出的是，如果多个节点同时发生故障，那么对于大型集群来说，随机副本放置有很高的数据丢失概率。我们计划基于最近提出的Copyset副本放置方案来解决这一挑战，我们计划开发有效的自适应策略，以最大限度地减少数据丢失概率，同时允许快速恢复。备份管理将使用超级对等覆盖网络来实现，考虑到不同的指标，包括负载和正在进行的恢复以及重新复制。

项目成果

High Throughput Log-Based Replication for Many Small In-Memory Objects

针对许多小型内存对象的高吞吐量基于日志的复制

• DOI: 10.1109/icpads.2016.0077
• 发表时间: 2016
• 期刊: 2016 IEEE 22nd International Conference on Parallel and Distributed Systems (ICPADS)
• 作者: Kevin Beineke;Stefan Nothaas;Michael Schöttner
• 通讯作者: Michael Schöttner

Fast Parallel Recovery of Many Small In-Memory Objects

• DOI: 10.1109/icpads.2017.00042
• 发表时间: 2017-12
• 期刊: 2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS)
• 作者: Kevin Beineke;Stefan Nothaas;M. Schöttner

Efficient Messaging for Java Applications Running in Data Centers

数据中心运行的 Java 应用程序的高效消息传递

• DOI: 10.1109/ccgrid.2018.00090
• 发表时间: 2018
• 期刊: 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)
• 作者: Kevin Beineke;Stefan Nothaas;Michael Schöttner
• 通讯作者: Michael Schöttner