Efficient Storage Systems for Real-Time Edge Computing

点击翻译按钮获取中文摘要

The data we produce is growing at an unprecedented pace. Interconnected Internet of Things (IoT) devices, including sensors, smartphones, and cameras are expected to generate 79.4 Zettabytes of data in 2025. In addition, a huge demand for real-time data processing will be posed by applications such as smart factories, health monitoring, augmented/virtual reality, and transportation. Real-time data (e.g., video, logs, location tracking) is predicted to be 30% of the data created in 2025. So far, we have been processing big data on powerful clusters in the cloud. Given the increasing need for real-time insights, cloud computing alone is no longer a viable solution. For instance, a traffic monitor in a smart city cannot wait for cameras' video footage to travel to a distant datacenter, to be processed, and sent back when controlling a busy intersection at rush-hour. Edge computing complements cloud computing, promising low latency and decreased bandwidth use. Broadly, the idea is to bring data processing to the data sources, by building micro datacenters close to the IoT devices. Edge computing is a growing area with countless practical use-cases and high potential for multi-disciplinary research. One of the main challenges of edge computing will be efficient data management. The long-term goal of this research is to develop efficient storage platforms for micro datacenters. Our approach involves innovation at the data-structures level and at the systems level. We leverage new storage technologies, in particular non-volatile memory and fast drives (e.g., Intel Optane SSDs). In the short/medium-term, we will focus on three objectives. First, we will design data-structures to handle heterogeneous data. For instance, a video stream should be easy to store alongside temperature measurements associated to the video frames. These data-structures also need to provide low-latency, high-throughput updates (prevalent in IoT workloads), and they should support efficient deletes because much IoT data is only useful within a short time window. Our second objective is building an efficient data pipeline. Naturally, data will flow between the edge devices, the micro datacenters, and the cloud. Here, the challenges lie in deciding on the right data tiering, while maintaining consistency across the layers of the storage system. Our final objective is system monitoring and robustness. Often, edge devices are located in remote areas with harsh conditions (e.g., on mountain tops). It is hence crucial that edge storage systems are self-healing, adaptable to workload changes without significant performance degradation, and easy to maintain with minimal human intervention on-site. We intend to open-source all the software we build, and aim to have impact both inside academia and in industry.