Advanced coding solutions for large-scale data storage & communication

动机*由于云存储、视频点播、直播和文件共享等在线服务的需求增加，全球IP流量快速增长。这些负责大部分互联网流量的在线服务严重依赖分布式存储系统(DSS)。因此，近年来，在世界范围内，DSS的数量和规模都在迅速增加。事实上，DSS已经是现代世界碳排放的最大贡献者之一。例如，在美国，它们仅次于航空业，预计到2020年将成为最大的贡献者。因此，对决策支持系统高效数据存储技术的研究引起了学术界和工业界的广泛关注。决策支持系统的硬件和软件故障会导致数据丢失和不可用。自然，许多服务提供商保留数据的多个副本以确保可靠性和可用性。当然，缺点是巨大的存储开销(通常是200%)和相关的维护成本。作为一种解决方案，最近提出了用于直扩系统的纠错编码。遗憾的是，大多数现有的纠错码不能用于这些大规模应用，因为它们产生的修复业务量超出了DS的网络容量。因此，人们强烈需要更高效的代码，为大规模应用程序量身定做。*目标*在此研究计划中，我们将利用我们过去对现代编码的知识和我们最近在DSS编码方面的活动，为分布式存储系统开发高效且可扩展的纠错编码解决方案。我们还将寻求不同性能指标(如编码开销、代码可靠性、修复流量、更新复杂性等)之间的基本关系的理论结果。*在这一领域有大量高影响的开放问题都适合培训HQP。*影响*本研究方案遵循了几个目标，这些目标对编码和信息理论领域的实践者和理论家来说都非常重要。提高大规模数据存储的效率不仅改善了我们的在线体验，还显著减少了DSS的碳足迹。我们希望我们通过这项工作取得的成果，以及我们与其他加拿大研究人员的贡献，将加拿大转变为该领域的主要科学中心，可以在技术和科学上支持位于加拿大的任何数据中心。事实上，由于冷却是数据中心的一大成本，加拿大因其较冷的气候和安全被认为是托管数据中心的理想地点。*在过去的十年里，世界上的在线活动显著增加。因此，大规模数据存储/通信目前是并将继续是一个巨大的工程挑战。我们在这一领域的活动将培训高需求量的HQP，并参与非常高影响的工程活动。

Motivation***The global IP traffic is growing rapidly due to increased demand for online services such as cloud storage, video on demand, live streaming, and file sharing. These online services, which are responsible for the bulk of the Internet traffic, heavily rely on distributed storage systems (DSS). Hence, around the world, the number and size of DSSs have rapidly increased in recent years. In fact, DSSs are already among the top contributors of carbon emission in the modern world. In the US, for example, they are only second to the airline industry and expected to be the top contributor by 2020. Thus, research on efficient data storage techniques for DSSs has recently attracted a lot of interest both in academia and in industry.***Background***Hardware and software failures in a DSS result in data loss and unavailability. Naturally, many service providers keep multiple copies of data for reliability and availability. The downside, of course, is the huge storage overhead (typically 200%) and the associated maintenance costs. As a solution, recently, error correction coding is suggested for DSSs. Unfortunately, most existing error correction codes cannot be used in these large-scale applications, because the repair traffic that they create is beyond the network capacity of the DSSs. Hence, a strong need for more efficient codes, tailored for large-scale applications, is felt. ***Goals ***In this research program, using our past knowledge of modern coding and our recent activities on coding for DSSs, we will develop efficient and scalable error correction coding solutions for distributed storage systems. We will also seek theoretical results on the fundamental relations among different performance measures (such as coding overhead, code reliability, repair traffic, update complexity, etc.). ***There are a large number of high-impact open problems in this area all suitable for training HQPs.***Impact***This research proposal follows several goals that are exceedingly important to both practitioners and theorists in the area of coding and information theory. Improving the efficiency of large-scale data storage not only improves our online experience, but also significantly reduces the carbon footprint of DSSs. We hope our results throughout this work and our contribution with other Canadian researchers turn Canada into a major scientific center in this field, which can technically and scientifically support any data center located in Canada. In fact, since cooling is a major cost in data centers, Canada because of its colder climate and its safety is considered as an ideal place to host data centers. ***In the past decade, the world has seen a significant increase in online activity. As a result, large-scale data storage/communication currently is and will continue to be a big engineering challenge. Our activity in this field will train HQPs that will be in high demand and involved in very high impact engineering activities.