Efficient Storage Systems for Real-Time Edge Computing

我们产生的数据正以前所未有的速度增长。到2025年，包括传感器、智能手机和相机在内的互联物联网（IoT）设备预计将产生79.4 zb的数据。此外，智能工厂、健康监测、增强/虚拟现实和交通等应用将对实时数据处理产生巨大需求。预计到2025年，实时数据（如视频、日志、位置跟踪）将占到数据总量的30%。到目前为止，我们一直在云端强大的集群上处理大数据。考虑到对实时洞察的需求日益增长，单独的云计算不再是一个可行的解决方案。例如，智能城市中的交通监控器在控制高峰时段繁忙的十字路口时，无法等待摄像头的视频片段传输到远程数据中心，进行处理并发送回来。边缘计算是云计算的补充，承诺低延迟和减少带宽使用。从广义上讲，这个想法是通过在物联网设备附近建立微型数据中心，将数据处理带到数据源。边缘计算是一个不断发展的领域，有无数的实际用例和多学科研究的巨大潜力。边缘计算的主要挑战之一将是有效的数据管理。本研究的长期目标是为微型数据中心开发高效的存储平台。我们的方法涉及数据结构层面和系统层面的创新。我们利用新的存储技术，特别是非易失性内存和快速驱动器（例如，英特尔Optane ssd）。在短期/中期，我们将重点关注三个目标。首先，我们将设计数据结构来处理异构数据。例如，视频流应该很容易与视频帧相关的温度测量一起存储。这些数据结构还需要提供低延迟、高吞吐量的更新（在物联网工作负载中很普遍），并且它们应该支持有效的删除，因为许多物联网数据只在短时间内有用。我们的第二个目标是构建一个高效的数据管道。当然，数据将在边缘设备、微型数据中心和云之间流动。这里的挑战在于决定正确的数据分层，同时保持存储系统各层之间的一致性。我们的最终目标是系统监控和健壮性。通常，边缘设备位于条件恶劣的偏远地区（例如，在山顶）。因此，至关重要的是，边缘存储系统能够自我修复，能够适应工作负载变化而不会出现显著的性能下降，并且易于维护，只需最少的现场人为干预。我们打算将我们开发的所有软件都开源，并希望在学术界和工业界都产生影响。

The data we produce is growing at an unprecedented pace. Interconnected Internet of Things (IoT) devices, including sensors, smartphones, and cameras are expected to generate 79.4 Zettabytes of data in 2025. In addition, a huge demand for real-time data processing will be posed by applications such as smart factories, health monitoring, augmented/virtual reality, and transportation. Real-time data (e.g., video, logs, location tracking) is predicted to be 30% of the data created in 2025. So far, we have been processing big data on powerful clusters in the cloud. Given the increasing need for real-time insights, cloud computing alone is no longer a viable solution. For instance, a traffic monitor in a smart city cannot wait for cameras' video footage to travel to a distant datacenter, to be processed, and sent back when controlling a busy intersection at rush-hour. Edge computing complements cloud computing, promising low latency and decreased bandwidth use. Broadly, the idea is to bring data processing to the data sources, by building micro datacenters close to the IoT devices. Edge computing is a growing area with countless practical use-cases and high potential for multi-disciplinary research. One of the main challenges of edge computing will be efficient data management. The long-term goal of this research is to develop efficient storage platforms for micro datacenters. Our approach involves innovation at the data-structures level and at the systems level. We leverage new storage technologies, in particular non-volatile memory and fast drives (e.g., Intel Optane SSDs). In the short/medium-term, we will focus on three objectives. First, we will design data-structures to handle heterogeneous data. For instance, a video stream should be easy to store alongside temperature measurements associated to the video frames. These data-structures also need to provide low-latency, high-throughput updates (prevalent in IoT workloads), and they should support efficient deletes because much IoT data is only useful within a short time window. Our second objective is building an efficient data pipeline. Naturally, data will flow between the edge devices, the micro datacenters, and the cloud. Here, the challenges lie in deciding on the right data tiering, while maintaining consistency across the layers of the storage system. Our final objective is system monitoring and robustness. Often, edge devices are located in remote areas with harsh conditions (e.g., on mountain tops). It is hence crucial that edge storage systems are self-healing, adaptable to workload changes without significant performance degradation, and easy to maintain with minimal human intervention on-site. We intend to open-source all the software we build, and aim to have impact both inside academia and in industry.