Efficient Storage Systems for Real-Time Edge Computing

我们生产的数据正以前所未有的速度增长。2025年，包括传感器、智能手机和摄像头在内的互联物联网(IoT)设备预计将产生79.4 ZB的数据。此外，智能工厂、健康监测、增强/虚拟现实和交通等应用将对实时数据处理提出巨大需求。实时数据(例如，视频、日志、位置跟踪)预计将占2025年创建的数据的30%。到目前为止，我们一直在云中强大的集群上处理大数据。鉴于对实时洞察的需求日益增长，仅靠云计算已不再是可行的解决方案。例如，智能城市中的交通监测器在高峰时间控制繁忙的十字路口时，不能等待摄像头的视频片段传输到遥远的数据中心进行处理，然后发送回来。边缘计算是云计算的补充，承诺低延迟和减少带宽使用。概括地说，这个想法是通过在物联网设备附近建立微型数据中心，将数据处理带到数据源。边缘计算是一个不断发展的领域，拥有无数的实际用例和多学科研究的高潜力。边缘计算的主要挑战之一将是高效的数据管理。这项研究的长期目标是为微型数据中心开发高效的存储平台。我们的方法涉及数据结构层面和系统层面的创新。我们利用新的存储技术，特别是非易失性存储器和快速驱动器(例如英特尔Optane固态硬盘)。在短期/中期，我们将重点关注三个目标。首先，我们将设计数据结构来处理异类数据。例如，视频流应该容易与与视频帧相关联的温度测量一起存储。这些数据结构还需要提供低延迟、高吞吐量的更新(在物联网工作负载中很常见)，并且它们应该支持高效的删除，因为许多物联网数据仅在短时间窗口内有用。我们的第二个目标是建立一条高效的数据管道。自然，数据将在边缘设备、微型数据中心和云之间流动。在这里，挑战在于决定正确的数据分层，同时保持存储系统各层之间的一致性。我们的最终目标是系统监控和健壮性。通常，边缘设备位于条件恶劣的偏远地区(例如，山顶)。因此，边缘存储系统必须具有自我修复能力，能够适应工作负载变化而不会显著降低性能，并且易于维护，而只需最少的现场人工干预。我们打算将我们构建的所有软件开源，并致力于在学术界和工业界产生影响。

The data we produce is growing at an unprecedented pace. Interconnected Internet of Things (IoT) devices, including sensors, smartphones, and cameras are expected to generate 79.4 Zettabytes of data in 2025. In addition, a huge demand for real-time data processing will be posed by applications such as smart factories, health monitoring, augmented/virtual reality, and transportation. Real-time data (e.g., video, logs, location tracking) is predicted to be 30% of the data created in 2025. So far, we have been processing big data on powerful clusters in the cloud. Given the increasing need for real-time insights, cloud computing alone is no longer a viable solution. For instance, a traffic monitor in a smart city cannot wait for cameras' video footage to travel to a distant datacenter, to be processed, and sent back when controlling a busy intersection at rush-hour. Edge computing complements cloud computing, promising low latency and decreased bandwidth use. Broadly, the idea is to bring data processing to the data sources, by building micro datacenters close to the IoT devices. Edge computing is a growing area with countless practical use-cases and high potential for multi-disciplinary research. One of the main challenges of edge computing will be efficient data management. The long-term goal of this research is to develop efficient storage platforms for micro datacenters. Our approach involves innovation at the data-structures level and at the systems level. We leverage new storage technologies, in particular non-volatile memory and fast drives (e.g., Intel Optane SSDs). In the short/medium-term, we will focus on three objectives. First, we will design data-structures to handle heterogeneous data. For instance, a video stream should be easy to store alongside temperature measurements associated to the video frames. These data-structures also need to provide low-latency, high-throughput updates (prevalent in IoT workloads), and they should support efficient deletes because much IoT data is only useful within a short time window. Our second objective is building an efficient data pipeline. Naturally, data will flow between the edge devices, the micro datacenters, and the cloud. Here, the challenges lie in deciding on the right data tiering, while maintaining consistency across the layers of the storage system. Our final objective is system monitoring and robustness. Often, edge devices are located in remote areas with harsh conditions (e.g., on mountain tops). It is hence crucial that edge storage systems are self-healing, adaptable to workload changes without significant performance degradation, and easy to maintain with minimal human intervention on-site. We intend to open-source all the software we build, and aim to have impact both inside academia and in industry.