Efficient Storage Systems for Real-Time Edge Computing

用于实时边缘计算的高效存储系统

• 批准号: RGPIN-2021-02662
• 负责人: Balmau, OanaMaria
• 金额: $ 1.75万
• 依托单位: McGill University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=750334
• 关键词: Efficient; Storage; Systems; Real; Time

The data we produce is growing at an unprecedented pace. Interconnected Internet of Things (IoT) devices, including sensors, smartphones, and cameras are expected to generate 79.4 Zettabytes of data in 2025. In addition, a huge demand for real-time data processing will be posed by applications such as smart factories, health monitoring, augmented/virtual reality, and transportation. Real-time data (e.g., video, logs, location tracking) is predicted to be 30% of the data created in 2025. So far, we have been processing big data on powerful clusters in the cloud. Given the increasing need for real-time insights, cloud computing alone is no longer a viable solution. For instance, a traffic monitor in a smart city cannot wait for cameras' video footage to travel to a distant datacenter, to be processed, and sent back when controlling a busy intersection at rush-hour. Edge computing complements cloud computing, promising low latency and decreased bandwidth use. Broadly, the idea is to bring data processing to the data sources, by building micro datacenters close to the IoT devices. Edge computing is a growing area with countless practical use-cases and high potential for multi-disciplinary research. One of the main challenges of edge computing will be efficient data management. The long-term goal of this research is to develop efficient storage platforms for micro datacenters. Our approach involves innovation at the data-structures level and at the systems level. We leverage new storage technologies, in particular non-volatile memory and fast drives (e.g., Intel Optane SSDs). In the short/medium-term, we will focus on three objectives. First, we will design data-structures to handle heterogeneous data. For instance, a video stream should be easy to store alongside temperature measurements associated to the video frames. These data-structures also need to provide low-latency, high-throughput updates (prevalent in IoT workloads), and they should support efficient deletes because much IoT data is only useful within a short time window. Our second objective is building an efficient data pipeline. Naturally, data will flow between the edge devices, the micro datacenters, and the cloud. Here, the challenges lie in deciding on the right data tiering, while maintaining consistency across the layers of the storage system. Our final objective is system monitoring and robustness. Often, edge devices are located in remote areas with harsh conditions (e.g., on mountain tops). It is hence crucial that edge storage systems are self-healing, adaptable to workload changes without significant performance degradation, and easy to maintain with minimal human intervention on-site. We intend to open-source all the software we build, and aim to have impact both inside academia and in industry.

我们生产的数据正在以前所未有的速度增长。互联的物联网（IoT）设备，包括传感器、智能手机和相机，预计到2025年将产生79.4泽字节的数据。此外，智能工厂、健康监测、增强/虚拟现实和交通运输等应用将对实时数据处理提出巨大需求。实时数据（例如，视频、日志、位置跟踪）预计将占2025年创建的数据的30%。到目前为止，我们一直在云中强大的集群上处理大数据。鉴于对实时洞察的需求日益增长，单独的云计算不再是一个可行的解决方案。例如，智能城市中的交通监控器无法等待摄像头的视频片段传输到远程数据中心，进行处理，并在高峰时段控制忙碌的十字路口时发送回来。边缘计算补充了云计算，有望降低延迟和减少带宽使用。从广义上讲，这个想法是通过在物联网设备附近建立微型数据中心，将数据处理带到数据源。边缘计算是一个不断增长的领域，拥有无数的实际用例和多学科研究的高潜力。边缘计算的主要挑战之一将是有效的数据管理。本研究的长期目标是为微型计算机开发高效的存储平台。我们的方法涉及数据结构级别和系统级别的创新。我们利用新的存储技术，特别是非易失性存储器和快速驱动器（例如，Intel Optane SSD）。中短期内，我们将重点实现三个目标。 首先，我们将设计数据结构来处理异构数据。例如，视频流应该易于与与视频帧相关联的温度测量一起存储。这些数据结构还需要提供低延迟、高吞吐量的更新（在物联网工作负载中很普遍），并且它们应该支持高效的删除，因为许多物联网数据只在很短的时间窗口内有用。 我们的第二个目标是建立一个高效的数据管道。当然，数据将在边缘设备、微计算中心和云之间流动。这里的挑战在于决定正确的数据分层，同时保持存储系统各层的一致性。 我们的最终目标是系统监控和鲁棒性。边缘设备通常位于条件恶劣的偏远地区（例如，在山顶上）。因此，边缘存储系统必须具有自我修复能力，能够适应工作负载变化而不会显著降低性能，并且易于维护，现场人为干预最少。我们打算开源我们构建的所有软件，并希望在学术界和工业界产生影响。